Skip to content

JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization

Sep 2026 · arXiv.org · Vol abs/2609.33601 · 0 citations · 61 references
Computer Science

TL;DR

JustQuant is proposed, a simple yet effective framework that moves the complexity of low-bit quantization from deployment-time operators into the training process and introduces Theseus QAD, a quantization-aware distillation method that progressively applies multi-level supervision, analogous to the gradual replacement process in the Ship of Theseus.

Abstract

Recent generative models have become increasingly powerful, but their inference cost continues to grow. Model quantization offers a promising way to compress these models and accelerate inference. However, at 4 bits, activation quantization is substantially more challenging than weight quantization. Recent post-training quantization (PTQ) and quantization-aware training (QAT) methods have made progress in 4-bit activation quantization by introducing smoothing, SVD branches, rotations, mixed precision, or advanced formats such as NVFP4. These additional operators and data types impose demanding requirements on inference engines and hardware, limiting the broad adoption of low-precision models. Can quantization be achieved using only plain low-bit operators? To answer this question, we propose JustQuant, a simple yet effective framework that moves the complexity of low-bit quantization from deployment-time operators into the training process. We first revisit model quantization from the perspective of knowledge distillation and show that a key reason existing PTQ and QAT methods fail is that they typically exploit supervision at only a single level. We then introduce Theseus QAD, a quantization-aware distillation method that progressively applies multi-level supervision, analogous to the gradual replacement process in the Ship of Theseus. Extensive experiments on DiT and diffusion large language models show two distinct regimes. For smaller models, Theseus QAD can serve as a lightweight warm-up stage that substantially improves subsequent QAT with plain operators, while naive QAD may collapse in the same setting. For larger models, Theseus QAD provides a stronger distillation training path than ordinary QAD. Across both regimes, JustQuant improves low-bit quantization quality while avoiding the complex operators required by many existing PTQ methods.

View source

Similar papers

#machine learning Preprint Sep 2026

Why Does Post-Training Quantization Work?

Comparing full-precision and quantized forward passes, and identifying two mechanisms that characterize pretrained quantization robustness, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and are verified across models and quantization set...

Yu-Xiang Chen, Michael Beyer, Jun Zhu et al. · 1 citation

ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs

ThinQuant is introduced, a data selection procedure which reduces the required number of calibration data points and an exact reduction of the associated optimization on this reduced calibration set using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant.

Mehdi Makni, Ryan Lucas, R. Mazumder · 0 citations
#machine learning Preprint Sep 2026

Quantization-Aware Pre-Training with Constrained Empirical Weight Distribution

Quantization-Aware Pre-Training (QAPT) can increase the inference efficiency of DNNs, but a problematic behaviour known as rounding boundary weight oscillation can introduce detrimental noise into the training process and significantly reduce convergence speed. While existing methods can reduce this detrimental noise,...

Ning-Feng Yang, T. Aamodt · 0 citations
Conference Open access Sep 2026

Quantization Stability Score: A Mixed-Precision Post-Training Quantization Method

This paper proposes Quantization Stability Score (QSS), a mixed-precision PTQ method that adaptively assigns bit-widths based on per-layer quantization stability, and demonstrates that QSS achieves superior accuracy-compression trade-offs.

Yu-Feng Zhao, Yi-Chi Zhang, Zhi-Yi Zhang · 0 citations
Oct 2026

Efficient zero-shot post-training quantization of large language models.

This work develops a fast, calibration-free, and uniform-precision quantization framework, and introduces two distinct methodologies: Hybrid Group Quantization (HGQ), which employs a Dual-Ruler strategy to statistically separate outliers from common weights, and Adaptive Grouped Vector Quantization (AGVQ), which learns...

Sami Ul Haq, A. El-Maleh, Ali Alsuwaiyan · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.