Skip to content

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Sep 2026 · 4 citations · 41 references
Computer Science

TL;DR

LeapQuant is proposed, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization and substantially reduces memory and compute costs during inference.

Abstract

Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and updating that state remains a major inference bottleneck. Quantization offers a natural way to reduce this cost, but can significantly degrade model quality, due to the accumulation of rounding errors and the presence of outlier rows and columns in the state. To address these challenges, we propose LeapQuant, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization. First, to mitigate error accumulation, we propose per-window quantization, which leaps over a window of tokens and quantizes the state only once at its end. Within a window, outputs are computed from the fixed low-bit state together with high-precision buffered updates. Second, to reduce the error introduced by each quantization, LeapQuant retains the state's largest outliers as a few high-precision Compensator Tokens, which share the update path of real tokens. We then smooth the remaining residual before quantization to further reduce the error. Comprehensive experiments across the Qwen, Kimi, and GLM model families show that LeapQuant substantially reduces memory and compute costs during inference. With accuracy comparable to the FP32 baseline, it achieves average speedups of 2.05--3.70$\times$ at the kernel level and 1.47$\times$ for end-to-end inference on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.

View source

Similar papers

#small language model Preprint Aug 2026

DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

DAMP uses both quantization-error energy and decay-based persistence to identify high-risk channels during offline calibration and stores these channels at higher precision and the remainder in INT8, the first to study post-training quantization of recurrent states in GDN and KDA based language models.

Tao Zhang, Jian-Chao Tan, Ping-Wei Sun et al. · 6 citations · ⚡3
#artificial intelligence Preprint Sep 2026

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

STEPQuant is proposed, a spatial-temporal post-training quantization framework for Delta-rule recurrent states that allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error.

Bing-Chen Yao, Hao-Bo Xu, Hao-Kun Lin et al. · 0 citations

Low-Discrepancy Dither for Quantized Recurrent State Caches

It is found that a deterministic golden-ratio Weyl dither consistently brings the quantized model closer to the full-precision one than stochastic rounding, across pure and hybrid models, storage formats, and long decoding horizons, at no extra cost.

Snigdha Chandan Khilar · 2 citations
#artificial intelligence Preprint Aug 2026

HyQuant: Hybrid-Precision Quantization for LLM Attention

Across diverse tasks, models, and datasets, HyQuant maintains nearly lossless accuracy with an extremely simple design, demonstrating the efficiency and practical feasibility of hybrid quantization for LLM attention.

Jiarui Ding, Bin Xing, Yu Zhang et al. · 0 citations
#artificial intelligence Preprint Sep 2026

From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers

This work defines one scalar loss over the joint Q, K, V weights of a block, evaluated against the block's real causally-masked attention output, and uses it twice: to fit the quantized weights, and to score the block for a multiple-choice knapsack allocation.

Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.