Evaluation under the recommended inference settings shows that Qwen3-4B achieves higher scores than Racka-4B, its Hungarian-adapted version, and testing the adapted models on a subset of the original benchmarks indicates performance decline in all three cases.
Abstract
In recent years, three initiatives have emerged to develop generative language models in Hungary. The motivation behind them is the same. For Hungarian, no model with the given capability existed, or existing English-centric models offered limited proficiency. A detailed examination of the corresponding studies, however, reveals several methodological limitations. First, the reliability of the evaluation protocols is questionable. Contrary to the findings of Csibi et al. [2026], evaluation under the recommended inference settings shows that Qwen3-4B achieves higher scores than Racka-4B, its Hungarian-adapted version. Data contamination is evident in the work of Yang et al. [2025d] and Szentmih\'alyi et al. [2025], potentially biasing the reported results. Second, the training pipelines fall short of current best practices in corpus curation and data mixture, which risks wasting substantial compute on low-quality data. The lack of controlled ablations prevents reliable assessment of these choices. Third, none of the three papers assessed forgetting or capability loss. Testing the adapted models on a subset of the original benchmarks indicates performance decline in all three cases, especially Racka-4B. These observations emphasize the importance of rigorous experimental design in language model development, given the significant computational and financial costs involved.
Despite rapid progress in evaluating large language models, the Lost in the Middle phenomenon - a persistent weakness of Transformer-based models whereby information near the middle of the context window exerts less influence on generation - remains unresolved. This paper examines the phenomenon in Serbian, among the f...
M. Jovićević, Slaviša Ilić, M. Gnjatović et al.· Zbornik radova· 0 citations
This paper proposes an approach to the quantitative assessment of Russian-language text generation quality using language models. An experimental analysis was conducted on structured messages generated by 14 language models differing in parameter count, architecture, and deployment conditions. The analysis employed the...
A. Y. Nosayev, V. V. Stepochkin, I. Y. Karyakin· Informatization and communic...· 0 citations
As language models (LMs) have improved, an increasingly common means of evaluating their linguistic competence is through explicit prompting (e.g.,"Which sentence is more acceptable?"). Prior work indicates that such prompting underperforms compared to standard token-probability-based methods in smaller models, but imp...
Corpus-based and experimental studies show that children acquiring non-Negative Concord (NC) languages often
display NC-like behaviour, producing or interpreting sentences with two negatives as a single negation. Previous work has taken
this as evidence that children initially posit a non-target NC grammar (e.g.,...
Multilingual evaluation benchmarks are the primary instrument for assessing whether large language models generalize beyond English, yet the adequacy of these benchmarks has received little systematic scrutiny. We present a data-driven audit of 51 recent multilingual benchmarks spanning 242 datasets and 219 languages,...
Manan Uppadhyay, Himanshu Beniwal, P. Kodali et al.· 1 citation