Skip to content
Open access

CultRAG at SemEval-2026 Task 7: Hybrid Sparse-Dense Retrieval with Entity-Centric Knowledge Bases for Cultural MCQ Answering

2026 · SemEval@ACL · pp. 3137-3142 · 1 citation · 11 references
Computer Science

TL;DR

The core finding is that RAG hurts rather than helps: the LLM-only baseline achieves 78.6% accuracy, outperforming the full system at 78.5% (McNemar’s test, p = 0 . 962).

Abstract

We present a trust-weighted Retrieval-Augmented Generation (RAG; Lewis et al., 2020) system for SemEval-2026 Task 7 (BLEnD) Track 2 (Ousidhoum et al., 2026), targeting English cultural multiple-choice QA across 30 countries. Built atop Llama-3.1-8B-Instruct (Meta AI, 2024), the six-phase pipeline integrates hybrid BM25+FAISS retrieval, country-aware filtering, intent detection, tiered routing, anti-leak prompt engineering, and trust-weighted reranking. The core finding is that RAG hurts rather than helps: the LLM-only baseline achieves 78.6% accuracy, outperforming the full system at 78.5% (McNemar’s test, p = 0 . 962 ). Oracle analysis reveals that only 40.7% of questions are answerable from the knowledge base, explaining why retrieval introduces more noise than signal. The sole recovery comes from anti-leak prompt filtering (Phase 4), which mitigates answer-anchoring artifacts. Code: https://github.com/CultRAG/ BLEnD-CultRAG .

Read PDF

Similar papers

Preprint Aug 2026

SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG

We introduce SciRet, a compute-aware empirical study of retrieval-augmented generation for scientific question answering over CORD-19. Rather than proposing a new model, we evaluate a fixed scientific RAG pipeline across three corpus scales: 1,034 chunks (1K papers), 5,160 chunks (5K papers), and 15,480 chunks (15K papers). The pipeline combines sentence-window chunking, BM25, BGE-M3 dense retrieval, reciprocal rank fusion, optional cross-encoder reranking, and grounded answer generation. Across these settings, hybrid retrieval is more robust than either sparse-only or dense-only retrieval in our setting, reaching Recall@10 of 1.000 at 1K and 15K. In contrast, an MS MARCO-trained cross-encoder reranker reduces precision on the scientific corpus, suggesting that domain mismatch can outweigh the benefits of stronger query-passage interaction. Generation faithfulness measured with RAGAS increases with corpus scale in our setup. Retrieval evaluation uses pseudo-relevance labels derived from the hybrid system, so we treat the results as controlled comparative evidence rather than a benchmark claim. We release code, indexes, and evaluation outputs to support replication and follow-up studies.

Kaysarul Anas Apurba, Mahade Hasan, Rofiqul Alam Shehab et al. · 0 citations
Book Open access Jul 2026

Effectiveness of Sparse Retrieval for Automated Fact-Checking

These findings establish learned sparse retrieval as a highly impactful design choice in automated fact-checking, with retrieval quality serving as a critical determinant of end-to-end performance in the studied systems.

Ritvik Setty, Vinay Setty · 0 citations
Preprint Jul 2026

Cross-Architecture LLM Ensembles, Feature-Based Reranking and Retrieval-Augmented Prompting for Legal Information Processing

Legal information processing spans retrieval, entailment and judgment prediction problems, requiring text matching, reasoning and robust generalisation with limited supervision. We report Team DU's participation in all five tasks of COLIEE 2026, using open-weight systems for legal case retrieval, case entailment, statute retrieval and entailment, and legal judgment prediction. For Tasks 3 and 4, all models predate the 15 July 2025 cutoff required by the rules. For Task 4 (statute entailment), a cross-architecture ensemble of nine models from three families achieves 96.3% accuracy, placing first among 33 submissions from 11 teams. For the Pilot Task (tort prediction and rationale extraction), a multi-view system combining five claim-level models and refining the verdict using features derived from the claim predictions achieves 73.1% TP accuracy and 68.2% RE F1 as an unofficial submission, scoring above all official entries on TP and matching the highest on RE. For Task 2 (legal case entailment), changing only the prompt from single- to multi-selection raises F1 from 0.343 to 0.555 in post-competition evaluation on released gold labels, exceeding the best official submission (F1 = 0.490). For Task 3 (statute retrieval and entailment), replacing the entailment model with Qwen3-235B and a structured legal reasoning prompt raises accuracy from 79.3% to 91.5% in post-competition analysis. For Task 1 (legal case retrieval), a learning-to-rank system combining lexical and semantic retrieval with structural, citation authority, and temporal features (34 in total) achieves F1 = 0.314 (rank 11 of 54 submissions from 22 teams). Overall, legal information processing benefits from different inductive biases across tasks, with cross-architecture ensembling, feature-based reranking and retrieval-augmented prompting each proving most effective in different settings.

Amal Saad Alshehri, N. Bencomo, Amir Atapour-Abarghouei · 0 citations
Review Open access Jul 2026

Retrieval-Augmented Generation for Trustworthy Enterprise LLM Assistants

This paper presents a technical synthesis of RAG for trustworthy enterprise assistants and describes the end-to-end pipeline in full: document chunking, embedding, vector indexing, retrieval, cross-encoder re-ranking, and grounded generation with inline citations.

Bini P B · 0 citations
Open access Jul 2026

ADAPTIVE MULTI-STAGE VECTOR RETRIEVAL FOR RETRIEVAL-AUGMENTED GENERATION

The Adaptive Multi-Stage Vector Retrieval (AMSVR) framework is proposed, prioritising weighted, drift-resistant composition over uniform fusion, and offers tailored configurations: AMSVR-Scientific (dense + tuned hybrid) peaks at NDCG@10 = 0.7570 on SciFact, while AMSVR-Full (seven stages) targets broader, noisier corpora where Recall@100 matters most.

Samsudeen Alabi Bankole, Yakub Kayode Saheed · 0 citations