Skip to content
Preprint

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

Aug 2026 · 2 citations · 58 references
Computer Science Engineering Mathematics

TL;DR

This paper derives closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model, and shows that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error in the transform domain.

Abstract

Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and serving capacity. The challenge is to reduce cache size while preserving the attention products, keeping reconstruction cheap, and using a fixed per-token bit count. At two bits per element, the most competitive methods rely on orthogonal transforms. However, existing techniques are either data-oblivious or use the query statistics without deriving the transform from a distortion criterion. Moreover, they rely on transforms built on top of random or Hadamard rotations, which equalize variances across entries rather than compacting energy, and fixed-width scalar quantizers, which are suboptimal at low rates. In this paper, we formulate KV cache quantization as a transform coding problem in which distortion is the error in the attention products. We derive closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model. We show that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error (MSE) in the transform domain. Thus, we can use MSE-optimal vector quantizers applied directly to the transformed key coefficients. To meet the fixed-width layout requirement, we show that grouping coefficients into equal-volume partitions makes equal-size codebooks attain the variable-rate optimum under the same high-resolution model. At two bits per element, our method, termed NOVA-KV, recovers most of the long-context retrieval accuracy lost by scalar quantization methods at comparable throughput.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Tailoring the Quantization Space for 1-Bit KV Cache Compression

The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwidth. To mitigate this bottleneck, vector quantization (VQ) has emerged as a promising approach for aggressive KV cache compression. However, existing VQ methods degrade sub...

Minsoo Cheong, Donghyun Son, Sungjoo Yoo · 2 citations · ⚡1
Preprint Aug 2026

KV Cache Compression Through the Lens of Transform Coding

Attention-Aware Transform Coding is introduced, which allocates bits over a calibration set to minimize attention-aware distortion and achieves near-lossless accuracy at approximately $5.8\times compression.

Hannah Laus, C. M. Verdun, Hao Wang et al. · 1 citation
#machine learning Preprint Sep 2026

WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms

This work introduces WUSH-KV, which adapts WUSH, which constructs a data-aware transform from the second-order statistics of both factors in a matrix product to reduce quantization error and achieves the lowest end-to-end perplexity among other tested transforms.

Jiale Chen, Vage Egiazarian, Eldar Kurtic et al. · 0 citations
#natural language process... Preprint Sep 2026

D-Quant: Driftable Entropy Coding for KV Cache Quantization

D-Quant is proposed, a flexible KV cache quantization framework that introduces a Drift mechanism to convert entropy-coded representations of each token into fixed-size bitstreams, enabling regular memory access and parallel dequantization within attention kernels.

Yi Su, Hong Liu, Guang-Hua Yu et al. · 0 citations

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

This work learns a selector that is able to produce, based on context, a per-layer cache configuration towards an overall compression budget, and achieves the Pareto frontier of accuracy against cache size, against learning-free and post-hoc baselines.

João Monteiro, Louis Béthune, A. Filippova et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.