Skip to content
Conference Open access

Adaptive Post-Training Quantization for Transformer-Based Language Models

Sep 2026 · 2026 International Conference on Sustainability, Innovation & Technology (ICSIT) · pp. 1-6 · 0 citations · 21 references

Abstract

Large language models are difficult to deploy on limited hardware because full-precision inference requires high memory, computation, and energy. This study evaluates adaptive post-training quantization as a practical compression approach that avoids full model retraining. Two strategies are examined: SmartQuant, which applies submodule-level sensitivity analysis and assigns higher weight precision to selected sensitive components, and Error-Guided Mixed Precision, which adapts group-level quantization decisions using module-wise quantization error. Experiments are conducted on LLaMA-3.2-1B and OPT-1.3B using WikiText2 and C4 perplexity, peak GPU memory, and quantization time. Among the tested settings, SmartQuant W6/W8-A16 remains closest to the FP16 baseline on both models while using fewer resources than the heavier EGMP configurations. EGMP W8A8 gives the lowest perplexity within its own settings but requires more memory and time. Sensitivity-aware mixed precision offers a practical quality-cost balance for compressing transformer-based language models under resource-constrained conditions.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.