Skip to content

Efficient zero-shot post-training quantization of large language models.

Oct 2026 · Neural Networks · Vol 206 Pt A, pp. 109750 · 0 citations · 29 references
Medicine

TL;DR

This work develops a fast, calibration-free, and uniform-precision quantization framework, and introduces two distinct methodologies: Hybrid Group Quantization (HGQ), which employs a Dual-Ruler strategy to statistically separate outliers from common weights, and Adaptive Grouped Vector Quantization (AGVQ), which learns local codebooks to capture sub-vector geometry.

Abstract

Recently, Large Language Models (LLMs) have revolutionized artificial intelligence; however, their massive parameter counts create a significant barrier to deployment on memory-constrained hardware. A critical challenge for ubiquitous AI is the memory bandwidth bottleneck, as billions of weights must be moved from memory to compute units for every generated token. While Post-Training Quantization (PTQ) offers a practical path to compression, existing state-of-the-art methods still struggle with major limitations: they either rely on computationally expensive calibration data, which risks overfitting or data unavailability, or use hardware-unfriendly mixed-precision representations to handle outliers. To address these limitations, this work develops a fast, calibration-free, and uniform-precision quantization framework. Specifically, we introduce two distinct methodologies: Hybrid Group Quantization (HGQ), which employs a Dual-Ruler strategy to statistically separate outliers from common weights, and Adaptive Grouped Vector Quantization (AGVQ), which learns local codebooks to capture sub-vector geometry. We test our methods across three diverse model families (Llama, OPT, and BLOOM), with model sizes ranging from 125M to 13B parameters. We outperform state-of-the-art calibration-based baselines in quantization runtime. HGQ reduces the quantization runtime for Llama-2-13B from over 30 minutes to under 20 seconds ( ∼ 100 ×  speedup) while maintaining competitive perplexity. Furthermore, AGVQ achieves highly competitive zero-shot accuracy on common sense reasoning tasks on the Llama-2-13B model, closely approximating the original FP16 performance while saving approximately 74% of the memory footprint. With fused reconstruction kernels, these savings further translate into measured speedup, HGQ reaching 1.28 ×  the decode throughput of the FP16 baseline on a 24 GB workstation GPU. These results demonstrate the potential of statistical, calibration-free approaches for democratizing accurate and efficient LLM deployment.

View source

Similar papers

#machine learning Open access Oct 2026

Trainable Vector Quantization for Large Language Models

Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference. However, extreme compression deteriorates model performance, which is challenging to recover due to the discrete nature of VQ parameters. Existing methods employ...

Anastasiia Prutianova, Alina Bogdanova, M. Shutov et al. · 0 citations

SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models

SPHQuant is a rotation-free spherical weight-only quantization framework for VLMs that isolates outlier magnitude into the radius while keeping directions bounded and statistically regular and designs a hardware-friendly GEMV kernel that keeps the direction codebook small enough for shared-memory lookup and packs radia...

Ke-Wei Zhang, Zheng Chen, Hao-Tong Qin et al. · 1 citation
Conference Open access Sep 2026

Adaptive Post-Training Quantization for Transformer-Based Language Models

Large language models are difficult to deploy on limited hardware because full-precision inference requires high memory, computation, and energy. This study evaluates adaptive post-training quantization as a practical compression approach that avoids full model retraining. Two strategies are examined: SmartQuant, which...

Abdurrahman M. Yusufi, A. Khobragade, Parth N. Mane et al. · 0 citations
Open access Oct 2026

LIRQuant: An Efficient Locally Isolated Rotation Quantization Method for Large Language Models

Post-Training Quantization (PTQ) enables efficient deployment of Large Language Models (LLMs), yet activation outliers remain a key challenge for low-bit quantization. Existing global transformation methods overlook heterogeneous channel sensitivity to quantization noise, leading to suboptimal accuracy-efficiency trade...

De-Sheng Xu, Hang Li, Tian-Chen Song et al. · 0 citations
Book Open access Nov 2026

EPTQ: Fast and Accurate 2-bit Post-Training Quantization via Factored E8 Lattice

Two-bit Post-Training Quantization (PTQ) offers an attractive path to deploying Large Language Models under strict memory budgets, yet scalar quantization suffers from catastrophic performance collapse at this extreme bit-width. While Vector Quantization (VQ) methods resolve the accuracy issue, existing approaches suff...

Ha-Myung Park, Jungeun Park, Chaewon Jeong et al. · 0 citations
Preprint Nov 2026

ProQuant: Progressive Quantization-aware Training for Edge MLLMs

Multi-modal large language models (MLLMs) with impressive perception and understanding capabilities will enable a range of downstream applications. However, their substantial computational and memory requirements hinder the real-world implementation, particularly for resource-constrained edge devices. Quantization has...

Yu-Fei Xue, Yu-Shi Huang, Jia-Wei Shao et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.