Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these systems using PR-AUC, a metric that only measures how well scores rank and ignores whether they are usable at a fixed threshold. We show this mismatch leads to systematically poor deployment choices, as models with the highest PR-AUC are often the worst in operation. We introduce Precision--Cache Hit Ratio (P-CHR) AUC, a cache-aware metric that measures precision across cache utilization levels, and Operational Retention Rate (ORR), which captures how much offline ranking quality survives at deployment. We decompose the operational gap between offline and deployed quality into a recoverable threshold-utility component and an irreducible structural component fixed by the dataset's positive rate. Our experiments show that the threshold-utility gap is governed by the training objective rather than data scale, and yields only to re-normalizing scores over the candidate pool or changing the training objective. Ultimately, model selection for semantic caching is a threshold-utility problem, not a ranking one, and measuring it is the first step to closing the gap.
Learning-augmented online algorithms use predictions to improve performance beyond worst-case guarantees while preserving robustness to prediction errors. In caching, existing approaches typically rely on signals such as next-arrival times or ranking scores, whose quality is measured through aggregate or worst-case err...
Helia Karisani, Mohammadreza Daneshvaramoli, Adam Lechowicz et al.· ACM SIGMETRICS Performance E...· 0 citations
Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-c...
Sam Wang, Julia White, Sahibzada Allahyar et al.· 1 citation· ⚡1
Major open-source serving stacks reuse the key-value tensors of a shared prefix by default. We measure what this prefix reuse costs in reproducibility. Holding model, decoding, seed and request order fixed at batch size one, we ran an eighty-episode multi-turn agentic tool-use workload with prefix reuse on and off acro...
This work compares the trade-offs between retrieval-augmented generation, prefix caching, and fine-tuning in large language models to show that when the knowledge is stable and matches the questions asked, fine-tuning is the greenest, fastest, and most accurate, because it produces shorter answers.
RaReCache is introduced, a framework that enables a large target model to decode accurately from a cache prefilled by a much smaller source via selective recomputation, establishing an efficient serving paradigm where small models prefill on behalf of massive targets, enabling large models to recompute only critical to...
Sreetama Sarkar, Saptarshi Mitra, Si-Tao Huang et al.· 0 citations
The cross-encoder study shows that thresholds do not transfer between embedding models, and LFU is the strongest simple default in this protocol; deployment decisions should first establish answer validity and then test sub-point policy differences with exact search.
Y. Kulkarni, Shubham Harkare, A. Babu· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
