A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
JoLT demonstrates that tensor-aware low-rank decomposition and quantized residuals, unified under a single storage budget, achieve near-lossless KV-cache compression across diverse model architectures without retraining.
Rahul G. Krishnan, Volker Schulz
· 0 citations