#natural language processing Preprint

SGD-KV: Summarization Guided KV Cache Compression

Ze-Yu Liu Woo-Min Song Xuandi Fu Sai Muralidhar Jayanthi V. Govindan A.G. Galstyan S. Bodapati S. Ronanki

Sep 2026 · 0 citations · 27 references

Computer Science

TL;DR

SGD-KV (Summarization-Guided KV Cache Compression), a head-aware framework that leverages a novel chunk-summarization diagnostic task to systematically identify and prioritize attention heads specialized in hierarchical information aggregation, is presented.

Abstract

Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooking the distinct functional roles of different attention heads. We present SGD-KV (Summarization-Guided KV Cache Compression), a head-aware framework that leverages a novel chunk-summarization diagnostic task to systematically identify and prioritize attention heads specialized in hierarchical information aggregation. Experiments on Qwen2.5-7B-1M and Qwen3-32B across diverse long-context benchmarks demonstrate that SGD-KV achieves state-of-the-art performance with contexts up to 1M tokens, while reducing KV cache memory usage by up to 75%. Our findings show that strategically allocating the KV cache budget based on the summarization score distribution of attention heads yields a superior efficiency-accuracy trade-off for long-context inference.

DOI arXiv Semantic Scholar

View source

Similar papers

#machine learning Preprint Sep 2026

Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference

The memory usage and decoding latency of LLM inference grow rapidly with context length. To reduce these costs, key-value (KV) cache compression methods selectively retain cached states based on token importance or differences in attention patterns across heads. However, we discover that retrieval capability varies sub...

Xian-Peng Shang, Can-Bin Huang, Jiang Li et al. · 0 citations

#machine learning Preprint Sep 2026

Cartridges++: KV Cache Compression without Off-Context Derailment

This study investigates the crucial deployment question of whether Cartridges and other CKVs can handle off-context queries, and proposes Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost.

Sonia Laguna, João Monteiro, Marco Cuturi et al. · 1 citation

#natural language process... Preprint Oct 2026

Behavior-Preserving KV Cache Compression

KV caches are a major bottleneck in long-context inference and long-form generation with large language models. Existing training-free eviction policies largely rely on proxy importance signals, such as attention mass, to decide which past tokens to retain. We argue that cache compression should instead preserve the pr...

Doo Hwan Hwang, Junyoung Jang, Jun-Ho Na et al. · 1 citation

Preprint Aug 2026

DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference

This work proposes DistillCache, a reinforcement learning framework that formulates KV-cache eviction as a sequential decision problem and demonstrates the effectiveness of learned, distribution-aware policies for memory-efficient long-context LLM inference.

Asaad Althoubi · 0 citations

Preprint Aug 2026

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

P PuzzleKV is proposed, a training- and calibration-free method that treats each completed page as an independent compression unit that achieves more than 96% of Full KV performance across both evaluated models and all benchmark settings, with substantial gains over Global SVD on RULER and competitive performance on Lo...

Zi-Zhong J. Wang, Jie-Ying Wang, Zhao Zhang et al. · 0 citations

#machine learning Preprint Aug 2026

SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

SemKV preserves every token, ranks tokens by a model-internal score, and assigns two adjacent above-cliff precisions, achieving a measured 6.0x storage reduction with no statistically detectable quality difference from full KV (n=900, three seeds), and outperforming FP16 token pruning granted a 1.5x larger memory budge...

D. Lee, Do-Hyung Kim, Jae-Hong Kim · 0 citations