LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and upda...
Yi Pan, Hao-Cheng Xi, Kan Zhu et al.
· 0 citations