Analysis of Reasoning-Intensive Retrieval in Japanese

Abstract

Reasoning-intensive retrieval has become increasingly important for Retrieval-Augmented Generation (RAG), where systems must retrieve evidence that supports an answer rather than rely on surface matching alone. However, existing benchmarks mainly focus on English and leave Japanese retrieval underexplored. We introduce JaReIR, a Japanese benchmark built from Yahoo! Chiebukuro question-answer pairs and Japanese Wikipedia passages with crowdsourced relevance and answerability judgments. JaReIR contains 153 queries split into Easy queries, which have at least one Direct answer passage, and Hard queries, whose positive passages all require reasoning. We benchmark sparse and dense retrievers on JaReIR and analyze the effects of query expansion and reranking. Performance generally drops from Easy to Hard across retrieval settings, which shows that Japanese reasoning-intensive retrieval remains challenging. Passage-level analyses and retrieved-title dispersion statistics show that Hard queries yield more ambiguous top-ranked results. Document-level retrieval reduces, but does not eliminate, the Easy-Hard gap, suggesting that passage localization explains part of the difficulty. Evaluating a translated Japanese version of an existing English benchmark under the same passage-level setup further suggests that translation noise and linguistic differences contribute to lower performance.