This paper derives closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model, and shows that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error in the transform domain.
Abstract
Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and serving capacity. The challenge is to reduce cache size while preserving the attention products, keeping reconstruction cheap, and using a fixed per-token bit count. At two bits per element, the most competitive methods rely on orthogonal transforms. However, existing techniques are either data-oblivious or use the query statistics without deriving the transform from a distortion criterion. Moreover, they rely on transforms built on top of random or Hadamard rotations, which equalize variances across entries rather than compacting energy, and fixed-width scalar quantizers, which are suboptimal at low rates. In this paper, we formulate KV cache quantization as a transform coding problem in which distortion is the error in the attention products. We derive closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model. We show that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error (MSE) in the transform domain. Thus, we can use MSE-optimal vector quantizers applied directly to the transformed key coefficients. To meet the fixed-width layout requirement, we show that grouping coefficients into equal-volume partitions makes equal-size codebooks attain the variable-rate optimum under the same high-resolution model. At two bits per element, our method, termed NOVA-KV, recovers most of the long-context retrieval accuracy lost by scalar quantization methods at comparable throughput.
The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwidth. To mitigate this bottleneck, vector quantization (VQ) has emerged as a promising approach for aggressive KV cache compression. However, existing VQ methods degrade sub...
Attention-Aware Transform Coding is introduced, which allocates bits over a calibration set to minimize attention-aware distortion and achieves near-lossless accuracy at approximately $5.8\times compression.
Hannah Laus, C. M. Verdun, Hao Wang et al.· 1 citation
This work introduces WUSH-KV, which adapts WUSH, which constructs a data-aware transform from the second-order statistics of both factors in a matrix product to reduce quantization error and achieves the lowest end-to-end perplexity among other tested transforms.
Jiale Chen, Vage Egiazarian, Eldar Kurtic et al.· 0 citations
D-Quant is proposed, a flexible KV cache quantization framework that introduces a Drift mechanism to convert entropy-coded representations of each token into fixed-size bitstreams, enabling regular memory access and parallel dequantization within attention kernels.
This work learns a selector that is able to produce, based on context, a per-layer cache configuration towards an overall compression budget, and achieves the Pareto frontier of accuracy against cache size, against learning-free and post-hoc baselines.
João Monteiro, Louis Béthune, A. Filippova et al.· arXiv.org· 1 citation
The results show that low-rank projection adaptation can recover held-out quality across fixed cache formats, while perplexity recovery need not restore long-context retrieval.
Seifeldin Abdellatif· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.