Skip to content

COMPARATIVE ANALYSIS OF THE APPLICABILITY OF LANGUAGE MODELS FOR GENERATING STRUCTURED MESSAGES IN RUSSIAN

2026 · Informatization and communication · 0 citations · 2 references

Abstract

This paper proposes an approach to the quantitative assessment of Russian-language text generation quality using language models. An experimental analysis was conducted on structured messages generated by 14 language models differing in parameter count, architecture, and deployment conditions. The analysis employed the following criteria: relevance, structure, linguistic quality, and instruction following. The generated outputs were evaluated in two ways: by three human experts and automatically using the “Large Language Model as a Judge” (LLM-as-a-Judge) method with strong language models Deepseek and GPT-5.1. The evaluation results can be considered reliable, as Kendall’s W = 0.925 indicated high agreement between the experts and the language models. It was found that generation quality depends nonlinearly on model scale: increasing the number of parameters does not always lead to improved results, and architectural optimizations and specialization can partially compensate for limited size. The obtained data allow us to determine the minimum model complexity level that ensures acceptable generation quality and can be used in the design of applied automatic text generation systems under limited computational resources.

View source

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.