Skip to content

Where Should a Document Live: Context, Representations, or Parameters?

Sep 2026 · 2 citations · 43 references
Computer Science

TL;DR

A controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks shows that in the oracle setting, Cartridges are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points.

Abstract

To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.

View source

Similar papers

Cartridges++: KV Cache Compression without Off-Context Derailment

This study investigates the crucial deployment question of whether Cartridges and other CKVs can handle off-context queries, and proposes Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost.

Sonia Laguna, João Monteiro, Marco Cuturi et al. · 1 citation
#artificial intelligence Preprint Sep 2026

An Empirical Study of VLM Pipelines for Long-Document QA

Three of the strongest pipelines succeed on different questions, and an oracle that picks the best pipeline per question gains roughly thirteen points over the best single pipeline, though evidence-type routing recovers almost none of it.

K. E. Ak, Jay Mohta, Gwang Lee et al. · 0 citations
Open access Sep 2026

Large language models as digital libraries: a multi-benchmark and multi-model study

Querying LLMs as digital libraries is feasible, but its effectiveness depends on model strength, deployment conditions, dataset structure, and execution strategy, and Galois remains valuable when relational discipline and controlled query execution are required.

Mirco Cazzaro, G. Silvello · 0 citations
Book Open access Aug 2026

NumCache: KV Cache Compression and Retrieval for Financial Document QA

The proposed NumCache, which compresses SEC filings into KV caches initialized from numerically dense regions and trained directly on financial QAs, is evaluated, which highlights cache-based retrieval with number-preserving representations as an effective approach for long-context financial QA.

Eftychia Makri, Peiwen Li, Yi-Dong Jiang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.