Open access

The Effects of Low-Bit Quantization on Confidence Calibration in Small Language Models

Gupta Iddhant

Jul 2026 · Indian Journal of Computer Science and Technology · 0 citations

Abstract

This study looks at how confidence patterns shift when cutting data precision to 4 bits using NF4, applied post-training on Phi-3.5-Mini-Instruct, a small-scale language model packing 3.8 billion parameters. Shrinking precision cuts down memory demands while speeding up output creation; however, what remains unclear is how such squeezing affects the way models rate their own sureness, measured via average prediction strength for each produced word unit. Rather than measuring correctness, the spotlight falls strictly on differences in how certain the system sounds across full-detail versus reduced-bit forms. Evaluation runs on ninety thoughtfully picked prompts split evenly among three kinds: factual assertions, fictional statements, and subjective stances. Hesitation creeps into quantized outputs across the board, their probability scores dipping five point seven six percent on average. Not every category bends the same way under pressure; made-up stories barely shift at all, while claims about reality sag a bit more. Opinions? Those take the hardest hit, certainty plummeting close to ten points lower. The same rules applied throughout, yet outcomes were split wide open based on what kind of knowledge was asked for. Oddly enough, false confidence shows up almost exclusively when facts are invented outright: the slimmed-down model gets things wrong but acts sure of itself, unlike its full-sized counterpart. This odd behavior earns the name “confident hallucination.” You might think shrinking models would show clear drops in accuracy or fluency, but standard metrics miss it completely. What matters instead is whether each guess lines up with the truth that matches how safely such systems can be used. Nowhere near steady, these dips shift with each kind of job. Shrinking a model can quietly erode reliability—something standard tests often miss. Where does it really show up? High-stakes areas rely on consistent results.

DOI Semantic Scholar

Read PDF

Similar papers

Preprint Jul 2026

Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation

This work measures, on identical inputs, how model scale and 4-bit quantization affect two confidence signals in the Qwen2-VL family: the confidence a model states in natural language, and its own mean token probability over the answer it generates, and argues that error-detection AUROC is the metric that exposes the difference between the two signals.

Asif Ferdous · 0 citations

Preprint Aug 2026

Which Decisions Low-Bit Quantization Breaks, and How to Predict Them

This work tracks quantization across 16 models from 8 families under round-to-nearest, seven under AWQ, two under GPTQ and one under GGUF, at 8 down to 2 bits, and measures the margin, the picked option's score minus its best alternative's, which removes the protection a large margin affords.

Zekun Wu, Swati Dhiman, Adriano S. Koshiyama · 1 citation

Preprint Jul 2026

Quantize with Confidence? An Empirical Study of Quantization for Code Generation

Six state-of-the-art quantization methods are evaluated on two representative large code model families using the multilingual McEval and CoderEval benchmarks for Python and Java to provide practical guidance for selecting quantization strategies for deploying large code models on resource-constrained hardware.

Saima Afrin, MD Zahidul Haque, A. Mastropaolo · 0 citations

Preprint Jul 2026

Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization

It is concluded that compression should not be treated as a way to remove memorized training data, and that extraction, not membership inference, is the number practitioners should be watching.

Akshay Sasi · 0 citations

Preprint Jul 2026

Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts

We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved. Using a six-category failure taxonomy validated by two independent human annotators (Cohen's $\kappa$ = 0.906), we classify 30,000 chain-of-thought outputs from five instruction-tuned LLMs (3B--14B parameters) across three quantization precisions (FP32, FP16, NF4) and four reasoning benchmarks. We find that while accuracy is robust across precisions (maximum 3.1 pp drop), Hollow Convergence (correct answers reached through incomplete or unverifiable reasoning) shows a significant size-dependent shift under NF4, dropping sharply for the two smallest models tested but remaining invariant for models at 12B parameters and above. This effect is also benchmark-specific: GSM8K is categorically immune while LogiQA and ARC-Challenge show the largest shifts. Furthermore, under NF4, Shortcut Collapse rises from 44% to 78% of wrong-answer failures in LLaMA 3.2-3B while Confidence Snowballing collapses from 15.8% to near zero, a qualitative shift invisible to accuracy metrics. Finally, we show Hollow Convergence cannot be reliably detected from surface-level text features (best F1 = 0.53), establishing it as a deployment-relevant failure mode that standard evaluation pipelines cannot catch.

Renuka Oladri, M. Priya, Jerry Wu · 0 citations

Preprint Jul 2026

QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

A benchmark that pairs a generative, multilingual stereotype probe with the refusal and multiple-choice controls that isolate open-ended generation, contrasts each build with and without reasoning, and rates the content severity of what it generates.

Emilio Ferrara · 0 citations