Oct 2026· Proceedings of the ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems· 0 citations· 23 references
TL;DR
This work presents a multi-provider benchmark of LLM-based metamodel-mapping suggestion, integrated in the Jjodel platform and run over eight canonical ATL Transformation Zoo transformations, with a ground truth reverse-engineered automatically from the hand-written .atl sources and a partial-match-aware comparison scheme.
Abstract
Specifying mappings between a source and a target metamodel is a recurring sub-task in model-to-model transformation, and large language models (LLMs) are increasingly proposed to suggest them inside modeling tools. Existing evidence, however, typically rests on a single provider, on manually curated ground truth, and on hit-or-miss scoring. We present a multi-provider benchmark of LLM-based metamodel-mapping suggestion, integrated in the Jjodel platform and run over eight canonical ATL Transformation Zoo transformations, with a ground truth reverse-engineered automatically from the hand-written .atl sources and a partial-match-aware comparison scheme. We evaluate GPT-4o, Claude Opus 4.6, and Mistral Large against a non-LLM EMFCompare baseline. All three providers suggest class-level mappings precisely and clearly outperform the baseline; feature-level quality is lower but does not collapse, with references harder than attributes. Because the matched mappings are almost all exact, the partial-match weight has little leverage and the provider ranking rests on exact matches; the providers’ self-reported confidence is well-aligned with correctness, letting a confidence threshold trade recall for precision. All artefacts are publicly available.
Software engineers need to identify and assess maintainability concerns in source code to support inspection and refactoring decisions. However, code-smell characterization is context-dependent: static-analysis tools such as SonarQube provide systematic rule-based findings, whereas LLMs offer contextual interpretation...
Diego Gatica, René Nöel· Proceedings of the ACM/IEEE...· 0 citations
N nondeterminism-aware evaluation for LLM-based MDE evaluation is introduced, which suggests that LLM-based MDE evaluation should treat correctness, reproducibility, and variation meaning as distinct dimensions.
Riccardo Rubei, Alessio Bucaioni, A. Di Salle· ACM/IEEE International Confe...· 0 citations
Large Language Models (LLMs) increasingly serve as reasoning components in modern software systems, yet a model that answers a logical query correctly can contradict itself when the same problem is restated in an equivalent form. Static benchmarks evaluate queries in isolation and therefore fail to expose such inconsis...
Large Language Models (LLMs) have shown promising performance in generating Object Constraint Language (OCL) constraints from natural language specifications. However, existing evaluations rely on publicly available UML models, which may overestimate generalization due to potential data leakage and reliance on recurrin...
Hamza Attarwala, Moataz Chouchen, Omar Alam et al.· ACM/IEEE International Confe...· 0 citations
Large language models (LLMs) have significantly advanced automatic code generation. Although they have demonstrated remarkable ability in writing standalone code, they still struggle with real-world repository-level coding tasks due to the lack of repository-specific knowledge. Existing studies typically employ retriev...
Tanghaoran Zhang, Xin-Jun Mao, Yu-Xin Zhao et al.· ACM Transactions on Software...· 0 citations
Findings show that detailed contextualization significantly improves the quality of LLM-generated suggestions and that more complex smells may benefit from an agentic approach.