The quant_eval Public Corpus: Behavioral Measurement of Quantization Degradation—Precision, Scale, and Substrate Effects

This paper presents the quant_eval public corpus: a paired behavioral evaluation of full-weight and quantized large language models across eight agent-relevant task families, published as eight open datasets with per-case evidence, paired statistical tests, and a verification chain a reader can check rather than trust. Quantization documentation reports memory footprint and perplexity. It does not report whether a quantized model still emits valid structured output, dispatches the right tool with the right arguments, carries state across turns, or completes a multi-step plan. This corpus measures those behaviors directly, per capability, against an identical full-weight baseline on byte-identical fixtures.

Method

Each run executes a full-weight lane and a quantized lane over the same 1,600 cases — 200 per family — with correctness defined by a deterministic first-fit oracle rather than a reference model. A family's pass rate is the fraction of cases satisfying every gating signal in that family's published conjunction. Paired differences are tested with McNemar's test, with a cluster adjustment published alongside the unadjusted value for the one family carrying semantic duplicates. A named taxonomy of harness artifacts is neutralized centrally so that output-format drift is not scored as capability loss. The evaluation methodology is quant_eval v7.22, a breaking redesign of the v7.21 framework described in the first quant_eval paper. Results from the two versions are not comparable. CORPUS Six runs across four base models, six model–precision pairs, and three quantization schemes. Mistral-Nemo-Instruct-2407 was evaluated at Q4_K_M, Q5_K_M, and Q8_0 against a shared F16 baseline, isolating precision. Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct-1M, and Qwen2.5-32B-Instruct were each evaluated at Q4_K_M, alongside Mistral-Nemo, isolating scale. 19,200 per-case rows in total.

Findings

The Mistral-Nemo full-weight baseline, executed three times across three runs, produced byte-identical outcomes on all 1,600 cases, establishing that the harness contributes no measurement noise on fixed hardware. Quantization cost is capability-specific and abrupt rather than gradual. At Q4_K_M, five of eight Mistral-Nemo families degrade significantly, two do not move, and one improves; at Q5_K_M and Q8_0, no family moves significantly in either direction. A single aggregate score would have averaged these outcomes into a number describing none of them. Quantization cost is model-dependent: at the same precision, Qwen2.5-7B loses two families where Mistral-Nemo loses five. The result cannot be transferred from one model to another; it has to be measured per model. On matched hardware, quantization delivered 1.71× to 2.70× faster wall time. Two runs showing apparent slowdowns are hardware-downgrade artifacts, not quantization effects, and are recorded as such in the published data. SCOPE AND

Limitations

The Qwen2.5-14B and Qwen2.5-32B runs were served through a remote inference endpoint originally deployed for a separate agent product. That endpoint declared no seed argument and fixed its sampling values and accelerator assignments at the application level, so those two runs are valid measurements under recorded conditions but are not bit-reproducible, and their wall-time ratios are not quantization effects. This is a property of that deployment, not of the hosting platform. Decoding temperature follows each publisher's model card and differs between the two model lineages, which confounds cross-model comparison. The paper states eight limitations explicitly, and names which published field lets a reader filter on each. The paper also accounts for what the measurement costs: under seven dollars of electricity and metered compute for the entire study, on a single consumer GPU, with capital stated separately. DATA AVAILABILITY All eight datasets are deposited under CC BY 4.0 and cited in the paper by concept DOI, with the corpus record at 10.5281/zenodo.22009419. Paired degradation statistics and family pass rates are recomputable in full from the published per-case rows; the paper states which datasets are recomputable and which are traceable through digests only. The derivation tooling is published under Apache-2.0. quant_eval itself is proprietary software of PBH Applied Systems, LLC and is not published.