COMPARATIVE ANALYSIS OF THE APPLICABILITY OF LANGUAGE MODELS FOR GENERATING STRUCTURED MESSAGES IN RUSSIAN
Abstract
This paper proposes an approach to the quantitative assessment of Russian-language text generation quality using language models. An experimental analysis was conducted on structured messages generated by 14 language models differing in parameter count, architecture, and deployment conditions. The analysis employed the following criteria: relevance, structure, linguistic quality, and instruction following. The generated outputs were evaluated in two ways: by three human experts and automatically using the “Large Language Model as a Judge” (LLM-as-a-Judge) method with strong language models Deepseek and GPT-5.1. The evaluation results can be considered reliable, as Kendall’s W = 0.925 indicated high agreement between the experts and the language models. It was found that generation quality depends nonlinearly on model scale: increasing the number of parameters does not always lead to improved results, and architectural optimizations and specialization can partially compensate for limited size. The obtained data allow us to determine the minimum model complexity level that ensures acceptable generation quality and can be used in the design of applied automatic text generation systems under limited computational resources.