STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through succes...
Bing-Chen Yao, Hao-Bo Xu, Hao-Kun Lin et al.
· 0 citations