Jul 2026· Annual International Computer Software and Applications Conference· pp. 468-477· 0 citations· 18 references
Abstract
Model-driven and low-code platforms rely heavily on domain-specific languages (DSLs) to capture structured system intent. While large language models (LLMs) have shown impressive capabilities in generating DSLs from natural language descriptions, proprietary models such as GPT-5 remain expensive, opaque, and often brittle in practice. This paper explores whether carefully orchestrated small language models (SLMs) can achieve comparable modeling quality while offering greater transparency, affordability, and control. We introduce a modular multi-agent framework in which small models take on specialized roles-including intent classification, DSL generation, syntactic validation, consistency checking, and automated repair. The framework supports two generation pipelines: one producing DSL code validated directly against a Lark grammar, and another generating structured JSON validated through Pydantic schemas. To address the inherent limitations of small models, we incorporate a lightweight prompt expansion step that enriches minimal user inputs with relevant domain concepts before DSL generation. We evaluated the approach across four experiments involving 30 different open-source SLMs. We measured syntactic correctness, structural similarity to GPT-5 outputs, alignment with expert-written DSLs, and overall modeling quality using independent LLM-as-a-judge evaluations. The results are compelling: feedback-driven repair dramatically increased the number of models producing valid DSLs (from 9 to 23), while prompt expansion improved structural similarity by 20-40 points and boosted F1 scores against expert references across all models. Blind evaluations by LLM judges consistently preferred the outputs from the expanded and repaired pipelines. Our findings demonstrate that architectural choices-such as prompt expansion, validation-feedback loops, and role-specialized agents-matter far more than raw model scale. This work shows that high-quality DSL generation is achievable with small, transparent models, opening the door to more accessible, costeffective, and trustworthy model-driven development platforms.
Large Language Models (LLMs) are predominantly assessed based on their common sense reasoning, language comprehension, and logical reasoning abilities. While models trained in specialized domains like mathematics or coding have demonstrated remarkable advancements in logical reasoning, there remains a significant gap in evaluating their code generation capabilities. Existing benchmark datasets fall short in pinpointing specific strengths and weaknesses, impeding targeted enhancements in models’ reasoning abilities to synthesize code.
To bridge this gap, this thesis introduces two novel contributions: CodeEval and CodeQual. CodeEval is an innovative, pedagogical benchmarking method that mirrors the evaluation processes encountered in academic programming courses. It comprises a multi-dimensional benchmark dataset of 602 hand-crafted problems designed to rigorously evaluate LLMs across 24 distinct aspects of Python programming, covering three proficiency levels—beginner, intermediate, and advanced—and includes both class-based and function-based problem types with detailed problem specifications and comprehensive test suites achieving 99.1% coverage. To facilitate widespread adoption, we developed RunCodeEval, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline. Our evaluation of 15 state-of-the-art LLMs revealed consistent performance degradation with increasing complexity (validated statistically, Cohen’s d = 0.790) and universal struggles with advanced concepts like concurrency.
Code quality is inherently subjective, encompassing dimensions like readability, efficiency, and adherence to language idioms that traditional static metrics fail to capture adequately. While large language models can assess these subjective qualities, lightweight models offer practical advantages: seamless CI/CD pipeline integration, lower operational costs, and full control over model behavior. We investigate whether such models can learn to assess code quality by training on synthetic LLM annotations. We introduce CodeQual, a dataset of 5,819 code samples derived from five established sources spanning diverse domains—competitive programming, pedagogical problems, software engineering, and general benchmarks—scored by LLMs across five quality dimensions, with 655 human-annotated samples for evaluation. Our fine-tuned model, CodeQualBERT, not only matches LLM performance but exceeds inter-human agreement on all five dimensions, achieving 16–100% improvement over the inter-human agreement baseline.
Together, these contributions provide a comprehensive framework for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
A test-driven pipeline is presented that extracts functional requirements (FR) from a problem description, resolves dependencies, maps them into a modular Model-View-Controller (MVC) structure, and generates tests before code, followed by bounded, execution-driven refinement.
Wasay Mohammed Abdul, Ragib Shahariar Ayon, Shibbir Ahmed et al.· SIGSOFT FSE Companion· 0 citations
BRIDGE is presented, a structured prompting framework that decomposes verification into three interconnected domains: Code (implementations), Specifications (formal intent), and Theorem State-ments (constructive correctness claims), and elicits domain-specific intermediate reasoning to connect them.
Robert Joseph George, Carson Eisenach, Udaya Ghai et al.· 0 citations
XREPOTEST is introduced, a multilingual repository-level benchmark for unit test generation spanning five underexplored languages: Rust, Go, Julia, PHP, and Ruby, and Invocation Rate is proposed to assess whether generated tests meaningfully exercise the intended functionality.
L. Dung, Dong Cao Van, Nam Le Hai et al.· 0 citations
This work proposes TraceDev, a multi-agent framework for automated software development grounded in use cases that contain multiple functional points and complex semantics, and demonstrates the effectiveness of TraceDev in repository-level code generation from requirements.
Mingyu Chen, Yakun Zhang, Zihao Xie et al.· 0 citations