PatchKV: Weight-Space Compensation of KV Cache
Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compression methods reduce the cache through token eviction or approximation, but degrade sharply at aggressive compression budgets. We propose PatchKV, a training-free framework...
Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi et al.
· 0 citations