Skip to content
Book Open access

Beyond Single-run Correctness: Nondeterminism-aware Evaluation of LLM-based Model Transformations

Oct 2026 · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems · 0 citations · 29 references
Computer Science

TL;DR

N nondeterminism-aware evaluation for LLM-based MDE evaluation is introduced, which suggests that LLM-based MDE evaluation should treat correctness, reproducibility, and variation meaning as distinct dimensions.

Abstract

Model transformation is a core model-driven engineering (MDE) operation in which reproducibility is expected: under fixed metamodels, source model, and transformation rules, a deterministic engine should produce a stable target model. Large Language Models (LLMs) are increasingly explored for MDE tasks, but evaluations often focus on whether an acceptable artifact can be produced once. For deterministic and quasi-deterministic MDE workflows, this single-run correctness view is necessary but insufficient. This paper introduces nondeterminism-aware evaluation for LLM-based MDE. Using model-to-model transformation as a stress case, we compare LLM-generated target models against deterministic ATL references and across repeated executions. Our exploratory evaluation covers five ATL Zoo transformation scenarios, four prompt configurations, three LLMs, and ten executions per transformation -scenario–configuration–LLM combination. We analyze reference deviation, inter-run variation, and morphological differences. The results show that transformation explicitness does not guarantee convergence. Even when the complete ATL transformation is provided, 13 out of 15 transformation-scenario–LLM combinations show non-zero mean and median deviation from the ATL reference, and 10 out of 15 exhibit non-zero inter-run variation. We further observe stable but not ATL-equivalent behavior and cases where a single exact match coexists with non-zero median deviation. These findings suggest that LLM-based MDE evaluation should treat correctness, reproducibility, and variation meaning as distinct dimensions.

Read PDF

Similar papers

Open access Oct 2026

STLC Manager: Benchmarking LLM-Driven Model-to-Test Transformation and Test Case Optimization

STLC Manager is presented, an LLM-supported framework that transforms requirements, UML/XML models, and source code into structured test scenarios and test cases and supports Serial, Bulk, and Parallel strategies for semantic redundancy reduction and shows greater configuration sensitivity for Parallel under the evalua...

Cem Bağlum, U. Yayan, Ahmet Yazıcı · 0 citations
Open access Nov 2026

WHEN CORRECT CODE IS NOT ENOUGH: ASSESSING THE PRODUCTION READINESS OF LLMGENERATED SOFTWARE

Large language models (LLMs) increasingly generate executable software, but functional correctness alone does not establish production readiness. This study evaluates GPT-5.6 Sol, Terra, and Luna across 15 algorithmic and production-oriented tasks using five generations per model-task pair, yielding 225 generated progr...

Ramesh Venkatachalapathy · 0 citations
Open access Oct 2026

CAST: A Compiler-Based Framework for Systematically Testing LLM Compositional Safety

This paper proposes CAST, a systematic testing framework designed to evaluate the compositional safety of LLMs in the domain of malicious code, and demonstrates that CAST systematically exposes severe safety violations in strongly aligned models that resist conventional red-teaming.

Lu Yan, Zhuo Zhang, Xiang-Zhe Xu et al. · 0 citations
Sep 2026

LLM-enabled Behavior Driven Development Workflow for Formally Verified Hardware Designs

This work proposes an integrated view on the use of LLMs for EDA and establishes an LLM-enabled behavior driven hardware development workflow, introducing and defining Formal Verification Gherkin Scenarios (FV Gherkin Scenarios), unlocking CNL specifications as the foundation for formally verified hardware designs via...

Luca Müller, Qian Liu, Rolf Drechsler · 0 citations
Book Open access Oct 2026

A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL

Large Language Models (LLMs) have shown promising performance in generating Object Constraint Language (OCL) constraints from natural language specifications. However, existing evaluations rely on publicly available UML models, which may overestimate generalization due to potential data leakage and reliance on recurrin...

Hamza Attarwala, Moataz Chouchen, Omar Alam et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.