A controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks shows that in the oracle setting, Cartridges are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points.
Abstract
To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.
This study investigates the crucial deployment question of whether Cartridges and other CKVs can handle off-context queries, and proposes Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost.
Sonia Laguna, João Monteiro, Marco Cuturi et al.· arXiv.org· 1 citation
Three of the strongest pipelines succeed on different questions, and an oracle that picks the best pipeline per question gains roughly thirteen points over the best single pipeline, though evidence-type routing recovers almost none of it.
K. E. Ak, Jay Mohta, Gwang Lee et al.· 0 citations
Querying LLMs as digital libraries is feasible, but its effectiveness depends on model strength, deployment conditions, dataset structure, and execution strategy, and Galois remains valuable when relational discipline and controlled query execution are required.
Mirco Cazzaro, G. Silvello· International Journal on Dig...· 0 citations
The number of experts used for query encoding can be reduced without retraining or re-indexing, retaining more than 99% of retrieval effectiveness while reducing query encoding time by up to 26%.
Anubhav Shrestha, Safal Shrestha, Minwu Kim et al.· 0 citations
The proposed NumCache, which compresses SEC filings into KV caches initialized from numerically dense regions and trained directly on financial QAs, is evaluated, which highlights cache-based retrieval with number-preserving representations as an effective approach for long-context financial QA.
Eftychia Makri, Peiwen Li, Yi-Dong Jiang et al.· Proceedings of the 32nd ACM...· 0 citations