Read What Matters: Query-Adaptive Quantization for KV Caches
KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places. We study this mismatch using separate budgets for retained bits and bits fetched per query. ReadKV stores each key and value in a progressive code whose prefixes support different reconstructi...
Siddharth Bhandari, Lucas Gretta, K. Balasubramanian et al.
· 0 citations