Skip to content

Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap

Sep 2026 · arXiv.org · Vol abs/2609.23742 · 2 citations · 16 references
Computer Science

TL;DR

It is found that CD eliminates all structural failures across all models, but content accuracy reveals a persistent semantic gap that is scale-dependent: type coercion failures are fully CD-rescuable, while instruction-semantic failures remain CD-resistant.

Abstract

Small open-source large language models (LLMs) in the 0.6B-4B parameter range are increasingly deployed for structured output generation (JSON, function calling, data extraction), yet little is known about how constrained decoding (CD) interacts with model scale in this regime. We benchmark five models from three families across 14 structured-output tasks under three decoding conditions (native, Outlines, XGrammar). We introduce a two-axis evaluation that separates structural correctness (schema validity) from semantic correctness (content accuracy). We find that CD eliminates all structural failures across all models (schema validity: 78.6-92.9% to 100%), but content accuracy reveals a persistent semantic gap that is scale-dependent: type coercion failures are fully CD-rescuable, while instruction-semantic failures (e.g., multi-step function calling) remain CD-resistant. Schema conformance is necessary but not sufficient for semantic correctness; CD's reach ends exactly where schema conformance ends.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

SemVerBench is introduced, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo), and six frontier models are evaluated: Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar).

Qi-Bai Chen, Ze-Ming Liu · 2 citations
Book Open access Oct 2026

SwiftGram: Dynamic Lexical Forking for Precise and Efficient Grammar-Constrained Decoding

Large Language Models (LLMs) must strictly adhere to grammatical constraints for reliable structured generation in agentic workflows, code synthesis, and domain-specific language (DSL) generation. Grammar-Constrained Decoding (GCD) enforces this by masking out tokens incompatible with the target grammar at each decodin...

Shu-Yi Ling, Liang-Yi Kang, Jie Liu et al. · 0 citations

On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Quality

Face/Off is introduced, a semantics-preserving identifier-renaming framework, and progressive naming conditions across multiple models and code-comprehension tasks are evaluated, revealing a systematic vulnerability in how current LLMs balance lexical cues against program structure.

Xin-Peng Shen, San-Zhuo Xi, Ya-Li Du et al. · 0 citations

The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models

System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA)...

Muhammad Usama, Dong Eui Chang · 0 citations

Probing Instruction Execution Stability of Large Language Models under Semantic Paraphrase

Analysis of instruction-preserving paraphrases shows that execution robustness is a distinct, model-dependent property not captured by accuracy alone and highlights the importance of evaluating prompt behavior under paraphrase in structured-output settings.

Bodhisatta Maiti, Debshree Chowdhury · 2 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.