DSL-Bench is presented, an executable benchmark and analysis framework for cross-DSL code generation and capability profiles and DSL-Profiler summaries for interpreting data-scarce DSLs are derived.
Abstract
Large language models (LLMs) are increasingly used in software-engineering workflows, but code-generation benchmarks largely focus on general-purpose languages. Domain-specific languages (DSLs) have distinct syntax, semantics, and validation requirements, making results from individual DSL benchmarks difficult to compare. We present DSL-Bench, an executable benchmark and analysis framework for cross-DSL code generation. Starting from 13,045 DSL-related papers, LLM-assisted open card sorting identifies 18 representative application domains. For each domain, we select an evaluable DSL and provide a syntax checker and task-level semantic oracle. Evaluating 7 LLMs on 18 DSLs, we observe mean syntax and semantic pass rates of 31.37% and 11.64%, respectively, revealing a substantial gap between syntax acceptance and semantic correctness. A variance decomposition attributes 76.99% of semantic pass-rate variation to differences among DSLs and 7.86% to differences among models. Among four DSL-level factors, Knowledge Demand has the strongest negative association with semantic correctness (r = -0.508). We evaluate prompt engineering across all 18 DSLs and study five improvement methods on five targeted DSLs. Prompt engineering yields uneven gains; RAG-Refinement, the strongest targeted method, improves semantic correctness by 16.29 percentage points on average over matched baselines. We further derive capability profiles and DSL-Profiler summaries for interpreting data-scarce DSLs. These results highlight toolchain-specific semantic bottlenecks and the need to evaluate DSL code generation beyond syntax acceptance.
STLC Manager is presented, an LLM-supported framework that transforms requirements, UML/XML models, and source code into structured test scenarios and test cases and supports Serial, Bulk, and Parallel strategies for semantic redundancy reduction and shows greater configuration sensitivity for Parallel under the evalua...
Cem Bağlum, U. Yayan, Ahmet Yazıcı· Applied Sciences· 0 citations
Large language models (LLMs) can generate register-transfer-level (RTL) code from natural-language specifications, but compilation alone does not establish structural completeness, functional correctness, or implementation efficiency. This paper presents a framework for evaluating LLM-generated RTL beyond compilation,...
Hepziba Susan, R. ShivaranjaniG, Malik Imran et al.· arXiv.org· 1 citation
Large language models (LLMs) have recently shown promise in automating formal specification generation, but their effectiveness has mainly been demonstrated on self-contained benchmark tasks. This paper examines the changes that occur when LLM-generated specifications are applied to an industrial verification project....
Du-Gang Liu, Ding Wang, Chao Jiang et al.· Proceedings of the 41st IEEE...· 1 citation
Large Language Models (LLMs) have transformed software engineering through automated code generation, yet their performance varies widely across target languages, specifically between mainstream (e.g., Python) and niche languages (e.g., OCL). Key challenges are determining which LLMs and prompting techniques can most e...
David Delgado· Proceedings of the ACM/IEEE...· 0 citations
Large language models (LLMs) are increasingly used for code generation, yet most evaluations still focus on whether generated code passes benchmark test suites. Functional correctness alone, however, provides only a limited view of quality: benchmark tests are often incomplete, and passing them does not guarantee that...
Jue-Zhe Huang, Tarek Mahmud, Mei-Ru Che et al.· 0 citations
SWE-Flux is introduced, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs.
Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.