Oct 2026· ACM/IEEE International Conference on Model Driven Engineering Languages and Systems· 0 citations· 29 references
Computer Science
TL;DR
N nondeterminism-aware evaluation for LLM-based MDE evaluation is introduced, which suggests that LLM-based MDE evaluation should treat correctness, reproducibility, and variation meaning as distinct dimensions.
Abstract
Model transformation is a core model-driven engineering (MDE) operation in which reproducibility is expected: under fixed metamodels, source model, and transformation rules, a deterministic engine should produce a stable target model. Large Language Models (LLMs) are increasingly explored for MDE tasks, but evaluations often focus on whether an acceptable artifact can be produced once. For deterministic and quasi-deterministic MDE workflows, this single-run correctness view is necessary but insufficient. This paper introduces nondeterminism-aware evaluation for LLM-based MDE. Using model-to-model transformation as a stress case, we compare LLM-generated target models against deterministic ATL references and across repeated executions. Our exploratory evaluation covers five ATL Zoo transformation scenarios, four prompt configurations, three LLMs, and ten executions per transformation -scenario–configuration–LLM combination. We analyze reference deviation, inter-run variation, and morphological differences. The results show that transformation explicitness does not guarantee convergence. Even when the complete ATL transformation is provided, 13 out of 15 transformation-scenario–LLM combinations show non-zero mean and median deviation from the ATL reference, and 10 out of 15 exhibit non-zero inter-run variation. We further observe stable but not ATL-equivalent behavior and cases where a single exact match coexists with non-zero median deviation. These findings suggest that LLM-based MDE evaluation should treat correctness, reproducibility, and variation meaning as distinct dimensions.
STLC Manager is presented, an LLM-supported framework that transforms requirements, UML/XML models, and source code into structured test scenarios and test cases and supports Serial, Bulk, and Parallel strategies for semantic redundancy reduction and shows greater configuration sensitivity for Parallel under the evalua...
Cem Bağlum, U. Yayan, Ahmet Yazıcı· Applied Sciences· 0 citations
Large language models (LLMs) increasingly generate executable software, but functional correctness alone does not establish production readiness. This study evaluates GPT-5.6 Sol, Terra, and Luna across 15 algorithmic and production-oriented tasks using five generations per model-task pair, yielding 225 generated progr...
Ramesh Venkatachalapathy· International Journal of Sof...· 0 citations
This paper proposes CAST, a systematic testing framework designed to evaluate the compositional safety of LLMs in the domain of malicious code, and demonstrates that CAST systematically exposes severe safety violations in strongly aligned models that resist conventional red-teaming.
Lu Yan, Zhuo Zhang, Xiang-Zhe Xu et al.· Proceedings of the ACM on So...· 0 citations
This work proposes an integrated view on the use of LLMs for EDA and establishes an LLM-enabled behavior driven hardware development workflow, introducing and defining Formal Verification Gherkin Scenarios (FV Gherkin Scenarios), unlocking CNL specifications as the foundation for formally verified hardware designs via...
Large Language Models (LLMs) have shown promising performance in generating Object Constraint Language (OCL) constraints from natural language specifications. However, existing evaluations rely on publicly available UML models, which may overestimate generalization due to potential data leakage and reliance on recurrin...
Hamza Attarwala, Moataz Chouchen, Omar Alam et al.· ACM/IEEE International Confe...· 0 citations
Controlled Non-Determinism, a process-oriented approach that allows variation without letting it directly determine the authoritative process state, is introduced.
Paul Mandl, Peter Mandl, Martin Häusl· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.