KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in the authors' multi-turn evaluations, and is found particularly useful for dynamic model routing.
Abstract
Large language models represent context with a key-value (KV) cache. Caches are model-specific: for the same text, models with different architectures or weights produce incompatible representations. This makes it costly to switch models over a shared context: although the context has already been processed by one model, the incoming model must process it again to build its own cache. We introduce KV-Lingo, a method for translating the KV cache of a source model into one that can be read by a target model. KV-Lingo consists of a collection of linear maps, typically one per layer of the target model, that are applied independently on all tokens'key and value representations. We train these maps using distillation, minimising the divergence between the target model's predictions from its native cache and those from the translated cache. We consider several model pairs spanning multiple sizes and architectures, training one translator per pair on a generic text corpus. The resulting translators preserve strong downstream performance in both small-to-large and large-to-small transfers. Since a switch then costs a linear map and a single decoding step instead of a prefill, replacing re-prefill with cache translation reduces the time to first token after a model switch by 9.6x already on a 64-token prompt for Qwen models on an Apple M3 Ultra, and by up to 29x at 32k context length on an H100. These gains make KV-Lingo particularly useful for dynamic model routing: a context can be processed by one model and handed off to another only when needed, without re-prefilling the shared prefix. We finally show that KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in our multi-turn evaluations.
Results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate context mobility as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.
Yi Li, Dongming Jiang, Yi Zhao et al.· 2 citations
This study investigates the crucial deployment question of whether Cartridges and other CKVs can handle off-context queries, and proposes Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost.
Sonia Laguna, João Monteiro, Marco Cuturi et al.· arXiv.org· 1 citation
Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key-value (KV) cache by T. The larger cache limits how many sequences a GPU can decode at once and slows each decoding step, which reads the whole cache. We propose Hybrid La...
Yu-Han Chen, Si-Yuan Zhang, Nan Wang et al.· 3 citations· ⚡1
Hypernetworks that map a context directly to a LoRA adapter let a large language model carry that context in its weights, but prior work has demonstrated them only on base models of up to 14 billion parameters. We present the Internalizer, a state-of-the-art, portable Context-to-Parameter Mapping hypernetwork that gene...
Peter Devine, Nick Ryan, Benjamin Sirb et al.· 0 citations
Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding and output matrices increase memory usage and slow inference, notably for small-scale models. It is also wasteful as mo...
Franck Signe Talla, Hippolyte Pilchen, François Yvon et al.· Conference on Empirical Meth...· 0 citations
Declarative Attention is introduced, a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes:(full context),(a specific region), and(recent output only).
Namgyu Ho, Huzama Ahmad, Woosung Koh et al.· 1 citation
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.