A test-driven pipeline is presented that extracts functional requirements (FR) from a problem description, resolves dependencies, maps them into a modular Model-View-Controller (MVC) structure, and generates tests before code, followed by bounded, execution-driven refinement.
Abstract
Large Language Models (LLMs) show promise for synthesizing software directly from natural-language problem descriptions. However, LLM-based code synthesis remains unreliable: models may hallucinate features, generated tests and code may diverge, and repairs often require manual effort. We present a test-driven pipeline that extracts functional requirements (FR) from a problem description, resolves dependencies, maps them into a modular Model-View-Controller (MVC) structure, and generates tests before code, followed by bounded, execution-driven refinement. We evaluate execution reliability with Pass@1 and refine@k, and assess the faithfulness of FR extraction using NLI-based entailment.
Large Language Model (LLM) pipelines can generate semantically meaningful unit tests with high coverage, but generated tests often fail to compile or execute. Many frameworks therefore use LLM-driven repair loops that iteratively re-prompt on failures until validation succeeds, but these repairs can introduce issues such as hallucinating an invented class solely to make the test pass. In this paper, we study repair-induced issues and effects of prompt strategy across three Java projects. We build a dataset of LLM-generated broken-to-fixed test pairs, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, derive a taxonomy from 117 manually analyzed pairs (three anti-pattern categories, seven scenarios), and compare the performance of prompting strategies through repair rate and post-repair coverage.
Model-driven and low-code platforms rely heavily on domain-specific languages (DSLs) to capture structured system intent. While large language models (LLMs) have shown impressive capabilities in generating DSLs from natural language descriptions, proprietary models such as GPT-5 remain expensive, opaque, and often brittle in practice. This paper explores whether carefully orchestrated small language models (SLMs) can achieve comparable modeling quality while offering greater transparency, affordability, and control. We introduce a modular multi-agent framework in which small models take on specialized roles-including intent classification, DSL generation, syntactic validation, consistency checking, and automated repair. The framework supports two generation pipelines: one producing DSL code validated directly against a Lark grammar, and another generating structured JSON validated through Pydantic schemas. To address the inherent limitations of small models, we incorporate a lightweight prompt expansion step that enriches minimal user inputs with relevant domain concepts before DSL generation. We evaluated the approach across four experiments involving 30 different open-source SLMs. We measured syntactic correctness, structural similarity to GPT-5 outputs, alignment with expert-written DSLs, and overall modeling quality using independent LLM-as-a-judge evaluations. The results are compelling: feedback-driven repair dramatically increased the number of models producing valid DSLs (from 9 to 23), while prompt expansion improved structural similarity by 20-40 points and boosted F1 scores against expert references across all models. Blind evaluations by LLM judges consistently preferred the outputs from the expanded and repaired pipelines. Our findings demonstrate that architectural choices-such as prompt expansion, validation-feedback loops, and role-specialized agents-matter far more than raw model scale. This work shows that high-quality DSL generation is achievable with small, transparent models, opening the door to more accessible, costeffective, and trustworthy model-driven development platforms.
J. Baber, Gabriela González-Sáez, Nicolas Hili et al.· Annual International Compute...· 0 citations
: Model-Driven Software Engineering has long excelled at generating code from static structural models, yet the specification and generation of dynamic behavioral models remains a persistent challenge. Meanwhile, Large Language Models (LLMs) offer flexible, natural-language based code generation but suffer from non-determinism and hallucinations. This paper presents a semi-formal approach that bridges these two paradigms for behavioral modeling via UML state machines. We contribute a textual modeling language that captures the essential elements of UML state diagrams—states, transitions, events, guards, and entry/exit actions—alongside a deterministic code generator that transforms state machine models into Java code following the Gang of Four State design pattern. The language supports two complementary action annotation styles: direct code fragments for concise, self-contained actions, and natural language descriptions for semantically richer behavior to be completed by an LLM weaver. LLM involvement is deliberately scoped to small, well-constrained action bodies, reducing token consumption and non-determinism compared to fully LLM-based approaches. Validated through the Gumball Machine case study, correctness is confirmed by automated tests covering state and transition coverage criteria, and repeating the LLM weaving step produced consistent results across all runs. Compared to both classical UML tooling and fully LLM-based generation, the approach offers stronger determinism, better traceability, lower cognitive modeling effort, and reduced computational cost, while retaining the flexibility to express complex action behavior in natural language where formal specification would be unnecessarily burdensome.
Oliver Engling, Felix Schwägerl, Thomas Buchmann· Proceedings of the 21st Inte...· 0 citations
This tutorial introduces a reusable, end-to-end evaluation pipeline grounded in empirical software engineering practices, focusing on post-generation validation rather than prompt design, allowing for validating AI-generated code in modern development workflows.
BRIDGE is presented, a structured prompting framework that decomposes verification into three interconnected domains: Code (implementations), Specifications (formal intent), and Theorem State-ments (constructive correctness claims), and elicits domain-specific intermediate reasoning to connect them.
Robert Joseph George, Carson Eisenach, Udaya Ghai et al.· 0 citations
This paper addresses automated unit test generation with large language models (LLMs). LLM-based test generation has not yet attained a quality level sufficient for practical use in industry. Although LLMs often reproduce API syntax faithfully, they frequently disregard semantic usage constraints and execution-environment dependencies, leading to assertion failures, mock-related errors, and reference/resolution errors. A prior failure analysis of Java unit test generation using GPT-4o classified 2980 trials into eight failure patterns and identified three root-cause mechanisms: external context ignorance, internal context ignorance, and a syntax–semantics gap. Building on that analysis, this paper proposes a prompt design comprising three strategies: (1) making the execution state explicit in the generated test, (2) stating semantic constraints explicitly, and (3) injecting environment constraints prior to generation. In contrast to generic techniques such as few-shot learning or chain-of-thought prompting, each proposed strategy is tied to a specific root-cause mechanism, yielding a systematic design in which each rule is explicitly justified by its correspondence to a specific root-cause mechanism. Experiments on 298 methods with five models (GPT-4o, GPT-5, GPT-5.1-Codex, Claude Sonnet 4.5, and Gemini 2.5 Pro) show improved test execution success rates for every model, with absolute gains ranging from 1.1 to 21.1 percentage points (pp). Mock-related errors were reduced by 61.9%–99.2% relative to the baseline prompt, demonstrating effectiveness against the targeted failure patterns. Finally, conditions under which the strategies transfer to other code-generation tasks are discussed, along with limitations on their scope.