A benchmark that pairs a generative, multilingual stereotype probe with the refusal and multiple-choice controls that isolate open-ended generation, contrasts each build with and without reasoning, and rates the content severity of what it generates.
Abstract
Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is assumed harmless and its safety is rarely re-checked. We find its principal side effect is increased bias that standard safety evaluation misses. Holding the model, its training, and the prompts fixed, a quantized model still refuses harmful requests, still avoids over-refusing benign prompts, and still selects the unbiased multiple-choice answer. Yet asked an open-ended question, the same model volunteers stereotypes in all eight languages we probe, in roughly one in four open-ended answers under an independent judge (~24% to ~27% across the compression ladder): it passes every standard check and still reaches users measurably more biased. The selective gap is a robust finding; whether open-ended bias further increases with compression is less certain, sensitive to the judge that scores it. We address both with \textbf{QuantiBias}, a benchmark that pairs a generative, multilingual stereotype probe with the refusal and multiple-choice controls that isolate open-ended generation, contrasts each build with and without reasoning, and rates the content severity of what it generates. Across two backbone models (Qwen and Gemma), a five-family screen, and eight benchmarks, quantizers allocate their extra precision by capability data that carries no bias-prevention signal, and reasoning before answering roughly halves the effect on some families while doing nothing on others. A quantized build must be re-evaluated for open-ended bias, not only on the short-form safeguards it already passes.
Six state-of-the-art quantization methods are evaluated on two representative large code model families using the multilingual McEval and CoderEval benchmarks for Python and Java to provide practical guidance for selecting quantization strategies for deploying large code models on resource-constrained hardware.
Saima Afrin, MD Zahidul Haque, A. Mastropaolo· 0 citations
Post-training quantization to 4-bit weights is widely reported to be nearly lossless. We test this claim for multi-turn, tool-calling agents, where it now matters most. On $\tau^2$-bench, across two open-weight model families in dense and MoE variants and two domains (eight cells, 456 episodes each, at 16-, 8-, and 4-bit weights), quantization indeed looks free on the standard metric. No cell shows a score change that survives multiple-comparison correction, and in the cell that carries the largest process damage, equivalence testing bounds the change within $\pm$7.5 points. The process tells a different story. Quantization amplifies the failure the model already exhibits at full precision (tool-name hallucination in telecom, with the same directional trend in retail entity errors) by up to 2.5$\times$ in volume (+17.6 points per task), while creating essentially no new failures. The failure set is the same at every precision (rank correlation $\geq$ 0.94, 0.18% novel events). The score stays flat because the benchmark's ten-error budget absorbs the extra failures. Shrinking the budget to two errors re-exposes a score gap of 17 points, and it does so only in the one cell where quantization added error volume, exactly as the masking account predicts. A targeted error-repair prompt, run for five telecom models at every precision, removes the damage exactly and only where it lives. Both diagnostics, the per-channel error rate and success under a shrinking budget, come from logs benchmarks already collect; we suggest reporting them alongside task reward.
Jiwon Jang, Kisu Yang, Heuiseok Lim et al.· 1 citation
This study looks at how confidence patterns shift when cutting data precision to 4 bits using NF4, applied post-training on Phi-3.5-Mini-Instruct, a small-scale language model packing 3.8 billion parameters. Shrinking precision cuts down memory demands while speeding up output creation; however, what remains unclear is how such squeezing affects the way models rate their own sureness, measured via average prediction strength for each produced word unit. Rather than measuring correctness, the spotlight falls strictly on differences in how certain the system sounds across full-detail versus reduced-bit forms. Evaluation runs on ninety thoughtfully picked prompts split evenly among three kinds: factual assertions, fictional statements, and subjective stances. Hesitation creeps into quantized outputs across the board, their probability scores dipping five point seven six percent on average. Not every category bends the same way under pressure; made-up stories barely shift at all, while claims about reality sag a bit more. Opinions? Those take the hardest hit, certainty plummeting close to ten points lower. The same rules applied throughout, yet outcomes were split wide open based on what kind of knowledge was asked for. Oddly enough, false confidence shows up almost exclusively when facts are invented outright: the slimmed-down model gets things wrong but acts sure of itself, unlike its full-sized counterpart. This odd behavior earns the name “confident hallucination.” You might think shrinking models would show clear drops in accuracy or fluency, but standard metrics miss it completely. What matters instead is whether each guess lines up with the truth that matches how safely such systems can be used. Nowhere near steady, these dips shift with each kind of job. Shrinking a model can quietly erode reliability—something standard tests often miss. Where does it really show up? High-stakes areas rely on consistent results.
Gupta Iddhant· Indian Journal of Computer S...· 0 citations
It is concluded that compression should not be treated as a way to remove memorized training data, and that extraction, not membership inference, is the number practitioners should be watching.
Position bias in multiple-choice LLM evaluation is widely cited as a confound in capability comparisons, but published measurements rely on single answer-order shuffles whose results confound the bias signal with content-level noise and sampling stochasticity. I introduce inspect_permute, an open-source extension to the inspect_ai evaluation framework that runs exhaustive answer-order permutations per question and reports the chi-squared / Cramer V signature of position bias with bootstrap confidence intervals. I apply the tool across four vendors (gpt-4o-mini, claude-haiku-4-5, gemini-2.5-flash, grok-3) on five MMLU subjects, 24,000 API calls under temperature-0 generation, with falsifier predictions pre-registered via a public SHA-256 hash before half the data was observed. Position bias turns out to be statistically detectable only within a roughly 60-95% base-accuracy Goldilocks zone. Below it, processing-load dominance swamps subject-specific signal; above it, ceiling effects compress the variance below the chi-squared test resolution. Detectable cells separate into two mechanism types: monotone A-to-D decrease (processing_load, in low-tier models) and non-monotone D-drop (content_ambiguity, in a narrow capability band). Standard MMLU places every frontier-tier model above the detection band, so absence of signal there should be read as not measurable, not unbiased. Together with the ceiling-effect characterisation in arXiv:2606.26185, this work brackets the detectable region of position-bias measurement and makes the field central question askable in a verifiable form. Package, data, preregistration under MIT.
This work tracks quantization across 16 models from 8 families under round-to-nearest, seven under AWQ, two under GPTQ and one under GGUF, at 8 down to 2 bits, and measures the margin, the picked option's score minus its best alternative's, which removes the protection a large margin affords.
Zekun Wu, Swati Dhiman, Adriano S. Koshiyama· 1 citation