KV-Lingo: Learning KV-Cache Translators with Distillation
KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in the authors' multi-turn evaluations, and is found particularly useful for dynamic model routing.
Valérie Castin, Keitaro Sakamoto, A. Filippova et al.
· 1 citation