A three-vector algorithm for detecting Russian-language neural-network-generated text fragments based on probabilistic analysis

Abstract

Objectives. The purpose of the work is to develop and programmatically implement a three-vector algorithm for detecting Russian-language neural network text fragments based on probabilistic analysis. The object of the study is Russian-language texts, the subject is statistical signs of their origin. Special attention is paid to the quantitative comparison of the proposed approach with the basic single-vector detection methods the perplexy detector and the GLTR rank analysis method. Methods. The proposed algorithm combines three feature vectors: token predictability, the proportion of rare lexemes, and the information density of the text, estimated via the algorithmic compression ratio. The author's regression language model rugpt3small is used as a probabilistic core. The implementation is made in Python using the transformers, PyTorch, NLTK, and zlib libraries. The experimental validation was conducted on a sample of 100 mixed documents (16,000 sentences) of five thematic groups, the neural network fragments of which were generated by four language models DeepSeek, GPT, Qwen and YandexAI. The quality was assessed by the deviation from the reference proportion of the neural network text and by classification metrics. Results. After applying the calibration adjustment, the average index for the sample was 45.2 points against a reference level of 50, and the values for all four models were in the narrow range of 44.445.9 points, which indicates the algorithm's stability to the choice of a generative model. The deviation of the average index from the standard was 4.8 points versus 11.1 points for the method based on perplexity and 7.6 points for GLTR; the F1 measure at the sentence level reached 0.76 versus 0.63 and 0.66, respectively. Conclusion. The three-vector algorithm has proven its effectiveness in the task of detecting neural network fragments and can serve as the basis for tools for verifying academic texts and anti-plagiarism systems. Further development of the research is related to the expansion of the sample, testing the resistance to paraphrasing and editing, as well as conducting ablative analysis