Reasoning-Aware Utility Estimation for Retrieval-Augmented Generation
Abstract
Retrieval-Augmented Generation (RAG) systems increasingly rely on retrieved documents to ground large language model outputs, yet evaluating whether those documents are truly useful for generation remains an open challenge. Recent work has shown that retrieval quality measured by conventional relevance-based metrics does not necessarily align with end-to-end RAG performance: a document judged relevant to a query may not help the model answer correctly, while an irrelevant document may actively distract generation[6, 18, 24]. Existing utility-aware evaluation methods address this limitation by estimating the utility of retrieved documents through isolated document-level probes. However, these approaches estimate document utility independently of the reasoning process. We argue that a document is useful not because it can answer a question in isolation, but because it contributes to the reasoning process that produces the final answer. To capture this behavior, we introduce Trace Utility Gain (TUG), a family of retrieval evaluation metrics that estimate document utility directly from chain-of-thought (CoT) reasoning traces generated over the full retrieval context. Specifically, TUG measures utility through three complementary signals: citation behavior, semantic alignment between retrieved documents and reasoning traces, and model self-assessment of document reliance. By estimating utility from the model's observed reasoning process, TUG captures how retrieved evidence contributes to answer generation rather than how useful it appears in isolation. We evaluate TUG on five question-answering benchmarks using three instruction-tuned large language models. Results show that TUG consistently achieves stronger correlation with end-to-end answer correctness than both traditional retrieval metrics and prior utility-aware approaches.