It is found that CD eliminates all structural failures across all models, but content accuracy reveals a persistent semantic gap that is scale-dependent: type coercion failures are fully CD-rescuable, while instruction-semantic failures remain CD-resistant.
Abstract
Small open-source large language models (LLMs) in the 0.6B-4B parameter range are increasingly deployed for structured output generation (JSON, function calling, data extraction), yet little is known about how constrained decoding (CD) interacts with model scale in this regime. We benchmark five models from three families across 14 structured-output tasks under three decoding conditions (native, Outlines, XGrammar). We introduce a two-axis evaluation that separates structural correctness (schema validity) from semantic correctness (content accuracy). We find that CD eliminates all structural failures across all models (schema validity: 78.6-92.9% to 100%), but content accuracy reveals a persistent semantic gap that is scale-dependent: type coercion failures are fully CD-rescuable, while instruction-semantic failures (e.g., multi-step function calling) remain CD-resistant. Schema conformance is necessary but not sufficient for semantic correctness; CD's reach ends exactly where schema conformance ends.
SemVerBench is introduced, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo), and six frontier models are evaluated: Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar).
It is argued that behavioral evaluation understates what models encode, while probing alone overstates what they deploy, in a three-level evaluation framework measured on the same items under the same binary decision.
Zhen-Yan Lu, He Wang, Xiao-Hui Huang· Volume 1· 0 citations
Large Language Models (LLMs) must strictly adhere to grammatical constraints for reliable structured generation in agentic workflows, code synthesis, and domain-specific language (DSL) generation. Grammar-Constrained Decoding (GCD) enforces this by masking out tokens incompatible with the target grammar at each decodin...
Shu-Yi Ling, Liang-Yi Kang, Jie Liu et al.· Proceedings of the 41st IEEE...· 0 citations
Face/Off is introduced, a semantics-preserving identifier-renaming framework, and progressive naming conditions across multiple models and code-comprehension tasks are evaluated, revealing a systematic vulnerability in how current LLMs balance lexical cues against program structure.
Xin-Peng Shen, San-Zhuo Xi, Ya-Li Du et al.· arXiv.org· 0 citations
System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA)...
Muhammad Usama, Dong Eui Chang· arXiv.org· 0 citations
Analysis of instruction-preserving paraphrases shows that execution robustness is a distinct, model-dependent property not captured by accuracy alone and highlights the importance of evaluating prompt behavior under paraphrase in structured-output settings.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.