Skip to content
Conference Open access

Position bias in multilingual language models applied to Serbian

2026 · Zbornik radova · 0 citations · 6 references

Abstract

Despite rapid progress in evaluating large language models, the Lost in the Middle phenomenon - a persistent weakness of Transformer-based models whereby information near the middle of the context window exerts less influence on generation - remains unresolved. This paper examines the phenomenon in Serbian, among the first such studies outside English, evaluating three recent open-weight small language models on a filtered multi-document QA dataset. Each query contains a question and ten documents, only one of which holds the answer; its position is shifted from first to tenth place to measure the resulting change in accuracy. A dedicated evaluation procedure was developed to account for the morphological complexity of Serbian, combining string-based matching with a final LLM-based judgment stage (Claude Haiku) for cases obscured by grammatical case variation. Within this dataset and context length, a Cochran's Q test revealed no statistically significant accuracy degradation for any of the three models as the gold document approached the middle of the context. Larger differences were instead observed between the models: Gemma-4-26B-A4B (92.9%) and Qwen3.6-35B-A3B (91.9%) could not be distinguished statistically (p = 0.151), while GPT-OSS-20B (87.3%) was reliably weaker than both (p < 0.0001), suggesting that model choice may play a more important role than gold-document position.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.