Cost-Aware Query Routing in RAG: Empirical Analysis of Retrieval Depth Tradeoffs
TL;DR
Cost-Aware RAG (CA-RAG), a routing framework that solves each query individually by treating each query individually, demonstrates that a small but well-designed set of retrieval strategies combined with lightweight per-query routing can meaningfully reduce the cost and latency of LLM deployments without compromising answer quality.
Abstract
When a large language model (LLM) answers a question using retrieved documents, retrieval-augmented generation (RAG) is the standard approach. Retrieving more documents improves answer accuracy but increases cost and response time; retrieving fewer documents saves resources but may miss critical information. Most existing RAG systems sidestep this dilemma by applying the same retrieval setting to every query, regardless of how simple or complex the question is. This wastes budget allocation on easy questions and under-serves hard ones. This paper introduces Cost-Aware RAG (CA-RAG), a routing framework that solves this problem by treating each query individually. For every incoming question, CA-RAG selects the most suitable retrieval strategy from a fixed menu of four options, ranging from no retrieval to fetching the top k=10 most-relevant documents. The selection is driven by a scoring formula that balances expected answer quality against predicted cost and response time. The weights in this formula act as dials: adjusting them shifts the system toward speed, savings, or quality without any retraining. CA-RAG is built on Facebook AI Similarity Search (FAISS) for document retrieval, OpenAI gpt-4o-mini for generation, and text-embedding-3-small for dense retrieval embeddings. We evaluate CA-RAG on a benchmark of 28 queries. The router assigns different strategies to different queries, achieving 26% fewer billed tokens compared to always using heavy retrieval and 34% lower response time compared to always answering without retrieval, while maintaining answer-quality parity in both cases. Further analysis shows that most savings come from simpler queries, where heavy retrieval was unnecessary. All results are reproducible from logged comma-separated value (CSV) files. CA-RAG demonstrates that a small but well-designed set of retrieval strategies combined with lightweight per-query routing can meaningfully reduce the cost and latency of LLM deployments without compromising answer quality.