JustQuant is proposed, a simple yet effective framework that moves the complexity of low-bit quantization from deployment-time operators into the training process and introduces Theseus QAD, a quantization-aware distillation method that progressively applies multi-level supervision, analogous to the gradual replacement process in the Ship of Theseus.
Abstract
Recent generative models have become increasingly powerful, but their inference cost continues to grow. Model quantization offers a promising way to compress these models and accelerate inference. However, at 4 bits, activation quantization is substantially more challenging than weight quantization. Recent post-training quantization (PTQ) and quantization-aware training (QAT) methods have made progress in 4-bit activation quantization by introducing smoothing, SVD branches, rotations, mixed precision, or advanced formats such as NVFP4. These additional operators and data types impose demanding requirements on inference engines and hardware, limiting the broad adoption of low-precision models. Can quantization be achieved using only plain low-bit operators? To answer this question, we propose JustQuant, a simple yet effective framework that moves the complexity of low-bit quantization from deployment-time operators into the training process. We first revisit model quantization from the perspective of knowledge distillation and show that a key reason existing PTQ and QAT methods fail is that they typically exploit supervision at only a single level. We then introduce Theseus QAD, a quantization-aware distillation method that progressively applies multi-level supervision, analogous to the gradual replacement process in the Ship of Theseus. Extensive experiments on DiT and diffusion large language models show two distinct regimes. For smaller models, Theseus QAD can serve as a lightweight warm-up stage that substantially improves subsequent QAT with plain operators, while naive QAD may collapse in the same setting. For larger models, Theseus QAD provides a stronger distillation training path than ordinary QAD. Across both regimes, JustQuant improves low-bit quantization quality while avoiding the complex operators required by many existing PTQ methods.
A new LLM compression method that co-optimizes pruning and quantization in a unified framework that performs favorably against two-stage baselines, even when subjected to extreme quantization settings designed for ultra-efficiency.
Comparing full-precision and quantized forward passes, and identifying two mechanisms that characterize pretrained quantization robustness, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and are verified across models and quantization set...
Yu-Xiang Chen, Michael Beyer, Jun Zhu et al.· 1 citation
ThinQuant is introduced, a data selection procedure which reduces the required number of calibration data points and an exact reduction of the associated optimization on this reduced calibration set using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant.
Mehdi Makni, Ryan Lucas, R. Mazumder· arXiv.org· 0 citations
Quantization-Aware Pre-Training (QAPT) can increase the inference efficiency of DNNs, but a problematic behaviour known as rounding boundary weight oscillation can introduce detrimental noise into the training process and significantly reduce convergence speed. While existing methods can reduce this detrimental noise,...
This paper proposes Quantization Stability Score (QSS), a mixed-precision PTQ method that adaptively assigns bit-widths based on per-layer quantization stability, and demonstrates that QSS achieves superior accuracy-compression trade-offs.
This work develops a fast, calibration-free, and uniform-precision quantization framework, and introduces two distinct methodologies: Hybrid Group Quantization (HGQ), which employs a Dual-Ruler strategy to statistically separate outliers from common weights, and Adaptive Grouped Vector Quantization (AGVQ), which learns...
Sami Ul Haq, A. El-Maleh, Ali Alsuwaiyan· Neural Networks· 0 citations