A training-free, multi-level disambiguation framework that generates diverse candidate specifications, identifies structural disagreements across these axes, ranks them by informativeness, and produces targeted clarification questions that progressively narrow the configuration space is proposed.
Abstract
The scale and complexity of modern cloud infrastructure have made"Infrastructure-as-Code"(IaC) essential for managing deployments through declarative configurations. While large language models (LLMs) are increasingly used to generate IaC configurations from natural language, user requests are often ambiguous and underspecified. Unlike traditional code generation, it is costly and time-consuming to test IaC configurations during the synthesis, forcing the LLMs into an almost one-shot regime. We observe that ambiguity in IaC synthesis exhibits a compositional structure: configurations decompose into three axes (resources, topology, attributes) where higher-level decisions constrain lower-level ones. We propose a training-free, multi-level disambiguation framework that generates diverse candidate specifications, identifies structural disagreements across these axes, ranks them by informativeness, and produces targeted clarification questions that progressively narrow the configuration space. We further introduce Ambig-IaC, an expert-verified benchmark of 300 validated IaC tasks with ambiguous requests, and define evaluation metrics based on graph edit distance and exact attribute matching. Comprehensive experiments show that our method outperforms existing interactive clarification baselines, with gains that scale with the interaction budget and generalize across models. Extensive ablation studies and analyses further demonstrate its robustness for interactive IaC synthesis.
BRIDGE is presented, a structured prompting framework that decomposes verification into three interconnected domains: Code (implementations), Specifications (formal intent), and Theorem State-ments (constructive correctness claims), and elicits domain-specific intermediate reasoning to connect them.
Robert Joseph George, Carson Eisenach, Udaya Ghai et al.· 0 citations
The results indicate that front-loading code understanding into a reusable, codebase-specialised vector store enables small local models to deliver grounded and repository-specific answers, making the agent well suited as a privacy-preserving development tool for in-house scientific codebases.
Aaron Nobile, Andreas Adelmann, Mohsen Sadr· 0 citations
DAREBench (Deployment-Aware and Reliable Evaluation of Models as Agents), a benchmark designed to capture workload variation and support reliable agent evaluation, is introduced, suggesting that agent deployment and model selection should consider workload profiles, deployment mode, and accuracy--cost trade-offs rather...
Yu Liu, Zhi-Lin Liu, Zhi-Wei Yang et al.· 0 citations
SchemaGUI, a template-based benchmark for controllable GUI generation evaluation, synthesizing paired natural language instructions and deterministic function-call references from parameterized interface schemas can generate thousands of deterministically annotated tasks in seconds without human labeling.
Jiarui Dong, Yin Cai, Zhouhong Gu et al.· 0 citations
LLM-based code generation fails when correctness depends on execution-dependent coupling: the meaning of one routine is defined by the runtime behavior of another, a relationship that cannot be resolved from textual descriptions alone. This limitation, which we call static binding, is not confined to explicitly coupled...
Gnaneswar Villuri, Hashmath Shaik, Alex Doboli· 1 citation
EvoGenUI-Bench is introduced, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios: information presentation, executable interaction, and tool-grounded external state.
Yue Peng, Lan-Ke Xia, Zi-Han Wang et al.· 1 citation