Skip to content
Book Open access

PosIR: A Multilingual Benchmark for Diagnosing Position Bias in Information Retrieval

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · 0 citations · 20 references

Abstract

Position bias—the systematic effect of where key information is located on retrieval effectiveness—is a critical concern in information retrieval. Although recent studies have identified such bias in dense retrievers, existing analyses focus predominantly on monolingual English, fail to disentangle document length from information position, and lack a standardized framework for controlled evaluation. To address these limitations, we introduce PosIR, a benchmark to systematically diagnose position bias across diverse retrieval scenarios, comprising 310 language–domain dataset combinations spanning 10 languages and 31 domains. Each query is paired with a positive document, a localized reference span, and positional metadata, enabling controlled analysis of retrieval quality across evidence locations and length bins. PosIR provides a complete data generation pipeline with quality-control filtering, along with baseline evaluation scripts. Experiments with 10 widely-used dense retrievers reveal that position bias is pervasive in both multilingual and cross-lingual retrieval, and that it intensifies as documents grow longer. PosIR serves as a foundational resource to advance the development of position-robust retrieval systems.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.