Skip to content
Book Open access

How Well Do LLMs Suggest Metamodel Mappings? A Multi-Provider Benchmark on Hand-Written ATL Transformations

Oct 2026 · Proceedings of the ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems · 0 citations · 23 references

TL;DR

This work presents a multi-provider benchmark of LLM-based metamodel-mapping suggestion, integrated in the Jjodel platform and run over eight canonical ATL Transformation Zoo transformations, with a ground truth reverse-engineered automatically from the hand-written .atl sources and a partial-match-aware comparison scheme.

Abstract

Specifying mappings between a source and a target metamodel is a recurring sub-task in model-to-model transformation, and large language models (LLMs) are increasingly proposed to suggest them inside modeling tools. Existing evidence, however, typically rests on a single provider, on manually curated ground truth, and on hit-or-miss scoring. We present a multi-provider benchmark of LLM-based metamodel-mapping suggestion, integrated in the Jjodel platform and run over eight canonical ATL Transformation Zoo transformations, with a ground truth reverse-engineered automatically from the hand-written .atl sources and a partial-match-aware comparison scheme. We evaluate GPT-4o, Claude Opus 4.6, and Mistral Large against a non-LLM EMFCompare baseline. All three providers suggest class-level mappings precisely and clearly outperform the baseline; feature-level quality is lower but does not collapse, with references harder than attributes. Because the matched mappings are almost all exact, the partial-match weight has little leverage and the provider ranking rests on exact matches; the providers’ self-reported confidence is well-aligned with correctness, letting a confidence threshold trade recall for precision. All artefacts are publicly available.

Read PDF

Similar papers

Book Open access Oct 2026

KatalIA: Can a Declarative Metamodel Make LLM-Based Smell Characterization Traceable?

Software engineers need to identify and assess maintainability concerns in source code to support inspection and refactoring decisions. However, code-smell characterization is context-dependent: static-analysis tools such as SonarQube provide systematic rule-based findings, whereas LLMs offer contextual interpretation...

Diego Gatica, René Nöel · 0 citations
Book Open access Oct 2026

Beyond Single-run Correctness: Nondeterminism-aware Evaluation of LLM-based Model Transformations

N nondeterminism-aware evaluation for LLM-based MDE evaluation is introduced, which suggests that LLM-based MDE evaluation should treat correctness, reproducibility, and variation meaning as distinct dimensions.

Riccardo Rubei, Alessio Bucaioni, A. Di Salle · 0 citations
Preprint Oct 2026

LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs

Large Language Models (LLMs) increasingly serve as reasoning components in modern software systems, yet a model that answers a logical query correctly can contradict itself when the same problem is restated in an equivalent form. Static benchmarks evaluate queries in isolation and therefore fail to expose such inconsis...

Zeng-Hui Zhou, Xiao-Ke Fang, Zheng Zheng · 0 citations
Book Open access Oct 2026

A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL

Large Language Models (LLMs) have shown promising performance in generating Object Constraint Language (OCL) constraints from natural language specifications. However, existing evaluations rely on publicly available UML models, which may overestimate generalization due to potential data leakage and reliance on recurrin...

Hamza Attarwala, Moataz Chouchen, Omar Alam et al. · 0 citations
Open access Oct 2026

LLM See, LLM Do: Enhancing Repository-Aware Code Generation in LLMs Through Logical Context and Usage Knowledge

Large language models (LLMs) have significantly advanced automatic code generation. Although they have demonstrated remarkable ability in writing standalone code, they still struggle with real-world repository-level coding tasks due to the lack of repository-specific knowledge. Existing studies typically employ retriev...

Tanghaoran Zhang, Xin-Jun Mao, Yu-Xin Zhao et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.