Oct 2026· Neural Networks· Vol 206 Pt A, pp.
109750
· 0 citations· 29 references
Medicine
TL;DR
This work develops a fast, calibration-free, and uniform-precision quantization framework, and introduces two distinct methodologies: Hybrid Group Quantization (HGQ), which employs a Dual-Ruler strategy to statistically separate outliers from common weights, and Adaptive Grouped Vector Quantization (AGVQ), which learns local codebooks to capture sub-vector geometry.
Abstract
Recently, Large Language Models (LLMs) have revolutionized artificial intelligence; however, their massive parameter counts create a significant barrier to deployment on memory-constrained hardware. A critical challenge for ubiquitous AI is the memory bandwidth bottleneck, as billions of weights must be moved from memory to compute units for every generated token. While Post-Training Quantization (PTQ) offers a practical path to compression, existing state-of-the-art methods still struggle with major limitations: they either rely on computationally expensive calibration data, which risks overfitting or data unavailability, or use hardware-unfriendly mixed-precision representations to handle outliers. To address these limitations, this work develops a fast, calibration-free, and uniform-precision quantization framework. Specifically, we introduce two distinct methodologies: Hybrid Group Quantization (HGQ), which employs a Dual-Ruler strategy to statistically separate outliers from common weights, and Adaptive Grouped Vector Quantization (AGVQ), which learns local codebooks to capture sub-vector geometry. We test our methods across three diverse model families (Llama, OPT, and BLOOM), with model sizes ranging from 125M to 13B parameters. We outperform state-of-the-art calibration-based baselines in quantization runtime. HGQ reduces the quantization runtime for Llama-2-13B from over 30 minutes to under 20 seconds ( ∼ 100 × speedup) while maintaining competitive perplexity. Furthermore, AGVQ achieves highly competitive zero-shot accuracy on common sense reasoning tasks on the Llama-2-13B model, closely approximating the original FP16 performance while saving approximately 74% of the memory footprint. With fused reconstruction kernels, these savings further translate into measured speedup, HGQ reaching 1.28 × the decode throughput of the FP16 baseline on a 24 GB workstation GPU. These results demonstrate the potential of statistical, calibration-free approaches for democratizing accurate and efficient LLM deployment.
Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference. However, extreme compression deteriorates model performance, which is challenging to recover due to the discrete nature of VQ parameters. Existing methods employ...
Anastasiia Prutianova, Alina Bogdanova, M. Shutov et al.· ECML PKDD· 0 citations
SPHQuant is a rotation-free spherical weight-only quantization framework for VLMs that isolates outlier magnitude into the radius while keeping directions bounded and statistically regular and designs a hardware-friendly GEMV kernel that keeps the direction codebook small enough for shared-memory lookup and packs radia...
Large language models are difficult to deploy on limited hardware because full-precision inference requires high memory, computation, and energy. This study evaluates adaptive post-training quantization as a practical compression approach that avoids full model retraining. Two strategies are examined: SmartQuant, which...
Abdurrahman M. Yusufi, A. Khobragade, Parth N. Mane et al.· 2026 International Conferenc...· 0 citations
Post-Training Quantization (PTQ) enables efficient deployment of Large Language Models (LLMs), yet activation outliers remain a key challenge for low-bit quantization. Existing global transformation methods overlook heterogeneous channel sensitivity to quantization noise, leading to suboptimal accuracy-efficiency trade...
De-Sheng Xu, Hang Li, Tian-Chen Song et al.· Conference on Empirical Meth...· 0 citations
Two-bit Post-Training Quantization (PTQ) offers an attractive path to deploying Large Language Models under strict memory budgets, yet scalar quantization suffers from catastrophic performance collapse at this extreme bit-width. While Vector Quantization (VQ) methods resolve the accuracy issue, existing approaches suff...
Ha-Myung Park, Jungeun Park, Chaewon Jeong et al.· Proceedings of the 35th ACM...· 0 citations
Multi-modal large language models (MLLMs) with impressive perception and understanding capabilities will enable a range of downstream applications. However, their substantial computational and memory requirements hinder the real-world implementation, particularly for resource-constrained edge devices. Quantization has...
Yu-Fei Xue, Yu-Shi Huang, Jia-Wei Shao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.