Jul 2026· International Conference on the Theory of Information Retrieval· 0 citations· 33 references
Computer Science
TL;DR
These findings establish learned sparse retrieval as a highly impactful design choice in automated fact-checking, with retrieval quality serving as a critical determinant of end-to-end performance in the studied systems.
Abstract
As fact-checking pipelines grow increasingly complex, a critical question remains unanswered: does retrieval matter more than reasoning? We investigate this question on the AVeriTeC 2.0 Shared Task, a leading and highly competitive benchmark for automated fact-checking that requires systems to verify real-world claims against open-domain web evidence. On this benchmark, we compare dense retrieval (FAISS), learned sparse retrieval (SPLADE), late interaction (ColBERT), and contextual embeddings across two architecturally distinct systems. SPLADE consistently outperforms all alternatives, improving new AVeriTeC scores by up to 86% on the development set and up to 27% on the hidden test set, where it enables a lower-ranked system to claim the top leaderboard position. Its advantage stems from vocabulary-space representations that bridge lexical gaps, preserve fine-grained numerical and entity matches, and remain robust to noisy web evidence, capabilities that are difficult to achieve with dense retrieval without additional techniques such as query expansion, reranking, or entity-aware training. Strikingly, two systems of vastly different complexity, one consolidating all reasoning into a single LLM call and the other orchestrating five models across seven tasks, converge to virtually identical performance once equipped with SPLADE. This demonstrates that retrieval quality is a major bottleneck in these systems. Beyond accuracy, SPLADE reduces preprocessing time by an order of magnitude while allowing both pipelines to share a single index. Our findings establish learned sparse retrieval as a highly impactful design choice in automated fact-checking, with retrieval quality serving as a critical determinant of end-to-end performance in the studied systems.
Automated fact-checking (AFC) systems retrieve evidence and predict claim veracity, yet evaluations omit simple baselines, systems are developed for a single benchmark and cannot be trusted to generalise across domains. No prior work cross-evaluates the full two-stage retrieve-then-verify pipeline across diverse datasets, complementing retrieval-only studies (Thakur et al., 2021) and single-stage benchmarking studies (Calamai et al., 2025). We benchmark nine models, ranging from random and sparse baselines to fine-tuned transformers, zero-shot LLMs, and the two highest-ranked systems from the AVeriTeC 2025 shared task, across four datasets spanning scientific, open-web, and climate domains. Three findings stand out: (1) on ClimateCheck claim-only and fine-tuned models outperform zero-shot LLM and top-performing AVeriTeC 2025 systems, highlighting that noisy evidence can degrade veracity prediction; (2) system rankings are strongly domain- and metric-dependent: the best model on SciFact (macro-F1 0.70) drops to 0.31 on ClimateCheck, while the AVeriTeC 2025 winner and runner-up swap rankings based on evaluation metrics and datasets; (3) replacing retrieved evidence with gold annotations improves veracity accuracy by 14-22 points across models, confirming retrieval remains primary bottleneck. We release code, pre-processed datasets, and all results to support reproducible AFC research.
Aida Usmanova, Z. Iklassov, Markus Leippold et al.· 0 citations
This work proposes MARS, a scalable knowledge graph question answering (KGQA) approach that requires no model fine-tuning, and performs a structured retrieval procedure that links question entities to the KG and iteratively retrieves relevant next-hop information.
Nikit Srivastava, Daniel Vollmers, René Speck et al.· 0 citations
On a human-annotated benchmark spanning eight datasets, Q-CARE achieves higher correlation with human judgments than four existing RAG evaluation metrics, including RAGEval and RAGChecker, proving its effectiveness as a reliable, automated evaluation framework.
Jeonghwan Choi, Taewon Yun, Minjeong Ban et al.· 1 citation
This work introduces TCR-Bench, a diagnostic benchmark for Table Content-level Answerability in RAG, and tests whether a lightweight two-stage pipeline, Answerability-Aware Reranking (AAR), applying direct query-table answerability judgment, can recover performance.
Jiaming Tian, Liyao Li, Wen-song Ye et al.· 0 citations
Reasoning-intensive retrieval has become increasingly important for Retrieval-Augmented Generation (RAG), where systems must retrieve evidence that supports an answer rather than rely on surface matching alone. However, existing benchmarks mainly focus on English and leave Japanese retrieval underexplored. We introduce JaReIR, a Japanese benchmark built from Yahoo! Chiebukuro question-answer pairs and Japanese Wikipedia passages with crowdsourced relevance and answerability judgments. JaReIR contains 153 queries split into Easy queries, which have at least one Direct answer passage, and Hard queries, whose positive passages all require reasoning. We benchmark sparse and dense retrievers on JaReIR and analyze the effects of query expansion and reranking. Performance generally drops from Easy to Hard across retrieval settings, which shows that Japanese reasoning-intensive retrieval remains challenging. Passage-level analyses and retrieved-title dispersion statistics show that Hard queries yield more ambiguous top-ranked results. Document-level retrieval reduces, but does not eliminate, the Easy-Hard gap, suggesting that passage localization explains part of the difficulty. Evaluating a translated Japanese version of an existing English benchmark under the same passage-level setup further suggests that translation noise and linguistic differences contribute to lower performance.
Retrieving relevant evidence from noisy web data is challenging, particularly in sensitive domains containing incomplete reports, heterogeneous language, and irrelevant content. We present Guardian Crawler, a reproducible retrieval-first testbed for controlled experiments on knowledge discovery and evidence-grounded summarization over synthetic web-like corpora. The architecture combines BM25 retrieval with risk-aware, embedding-augmented, and hybrid reranking, followed by constrained retrieval-augmented generation with explicit document citations. Experiments on a synthetic 900-document corpus and 10 queries produced the highest descriptive retrieval scores under risk-based reranking, with P@10 = 1.00 and NDCG@10 = 0.94, compared with 0.94 and 0.81 for BM25. The best hybrid and BM25+Semantic configurations reached NDCG@10 values of 0.94 and 0.88, respectively. All 41 evaluable generated bullets passed the lexical coverage threshold; an automated LLM judge classified 36 as supported, one as partially supported, and four as unsupported. These results demonstrate the feasibility of Guardian Crawler as a controlled testbed but do not establish statistical superiority, human-validated faithfulness, or transfer to live-web investigative environments.
J. Castillo, Santosh Nukavarapu, Ravi Mukkamala· 0 citations