QueryRoute is introduced, a benchmark that freezes the expensive artifacts needed to study this inference-time decision problem reproducibly: original queries, generated variants, ranked lists under multiple retrievers, retrieval scores, and per-query oracle labels.
Abstract
LLM-based query reformulation can improve retrieval, but no single reformulation strategy is consistently optimal across queries, domains, retrievers, or model backbones. This creates an inference-time decision problem: ``Given an original query and a pool of candidate reformulations, which one should be issued to the retriever?''. Existing studies are hard to compare because they use different reformulator pools, retrievers, relevance signals, training labels, and evaluation metrics. We introduce QueryRoute, a benchmark that freezes the expensive artifacts needed to study this decision reproducibly: original queries, generated variants, ranked lists under multiple retrievers, retrieval scores, and per-query oracle labels. The benchmark contains 3,757 queries, 11 candidate systems, five reformulator backbones, and three retrievers across TREC DL, BEIR, and BRIGHT, yielding 619,905 retrieval outcomes. We benchmark supervised classification, routing, QPP, and LLM-as-judge selectors. Results show substantial oracle headroom over fixed reformulators, but current selectors recover only part of it; selector rankings change across retrievers, and similar mean effectiveness can hide different query-level behavior. The released artifacts and evaluation harness allow future selectors to be compared without regenerating variants, rerunning retrieval, or rebuilding judge pipelines. Code and data are available at https://github.com/haisonle001/QueryRoute
This work systematically analyze how retriever and generator complexity interacts across factoid and multi-hop question answering (QA), including bridge and composition reasoning tasks, and introduces DRAG, a query-adaptive framework for selecting retriever-generator configurations.
Neeraj Anand, Payel Santra, Partha Basuchowdhuri et al.· 0 citations
It is found that although the queries triggering slow planning are largely DBMS-specific, recurring pathologies involving correlated subqueries, CTE expansion, repeated subquery expressions, disjunctive joins, and constant folding affect multiple systems.
Geoffrey X. Yu, Ryan Marcus, Tim Kraska· 0 citations
This approach is pre-retrieval and query-centric, making it robust in heterogeneous, case-like corpora and applicable across domains such as legal, healthcare, finance, and enterprise search.
Ye Xia, Emre Yamangil, Hai-Xun Wang· Proceedings of the 35th ACM...· 0 citations
Q2D-Web is introduced, a large-scale agentic retrieval benchmark consisting of a 190M-document web corpus and 70k agentic search queries in ten languages, reformulated from real-world user queries in production systems, and benchmark 13 retrievers including lexical, dense, and late-interaction models.
Maximilian Schall, Sedigheh Eslami, Markus Krimmel et al.· 0 citations
Pooled LLM evaluation is a practical and cost-effective workflow for incremental retrieval model selection in deployed systems, and 97% of pairwise system orderings are preserved once bootstrap uncertainty in the qrels is taken into account.
Max Nelson, Hanoz Bhathena, Aviral Joshi et al.· 0 citations
This work compares matched query-only and query+retrieval routers while holding the optional actions, router family, training procedure, and evaluation fixed and suggests retrieval-state features should not be credited with routing value unless they improve over a matched query-only control.