2026· Proceedings of the 21st International Conference on Software Technologies· 0 citations· 31 references
TL;DR
This paper investigates the reliability of LLMs in evaluating UML diagrams generated through reverse engineering processes (source code) and asks: do LLM assessments align with those of human experts?
Abstract
: Large Language Models (LLMs) are increasingly used to evaluate software engineering artifacts. This paper investigates the reliability of LLMs in evaluating UML diagrams generated through reverse engineering processes (source code). We ask: do LLM assessments align with those of human experts? A COMAS-based framework is proposed for trust-aware LLM selection. In the original COMAS, indirect trust is computed via Dijkstra max-product chains. We replace this with direct human–LLM comparisons for each criterion independently. This is valid since LLMs can evaluate any diagram on demand. Trust is measured using an L1-norm-based divergence that is robust under small-sample conditions. The developed framework supports flexible per-criterion LLM selection via expert-defined thresholds and weights. An empirical study involved 30 students evaluating five UML sequence diagrams. Four evaluation criteria were used: completeness of requirements description, non-contradiction to requirement specification, UML notation compliance, and Miller’s principle. Four LLMs were compared: Groq, Gemini, Mistral, and Copilot. For 90% of participants, a combination of two or three LLMs outperformed any single model. Hybrid per-criterion LLM selection is a viable strategy for automated UML diagram evaluation.
The proposed model aims to support the formalization of model selection processes, improve decision-making, and enhance the traceability and transparency of LLMOps practices and forms part of a broader research effort toward the formalization of the entire LLMOps life cycle.
Maria Chernigovskaya, A. Nahhas, Christian Haertel et al.· Proceedings of the 21st Inte...· 0 citations
This survey provides the first systematic synthesis of LLM-based diagram modelling research, highlighting needs for standardised benchmarks, stronger evaluation protocols, broader diagram coverage, and techniques for improving semantic reliability and multi-view consistency.
This work presents the first cross-task empirical evaluation of LLMs spanning five RE-related activities, as well as replication materials supporting reproducibility, and a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.
Jacek Dabrowski, Manjeshwar Aniruddh Mallya, Alessio Ferrari et al.· 0 citations
A lifecycle-aware framework that integrates quantitative software quality assessment with Large Language Model (LLM)-based code refinement is proposed and the potential of metric-driven LLM feedback for research software quality improvement is demonstrated while highlighting its inherently multi-objective nature.
Nafis Tanveer Islam, N. Soveizi, Yutong Li et al.· 0 citations
Assessing diagrammatic artifacts such as UML use‐case diagrams is a core yet challenging task in engineering and software engineering education. In large classes, manually evaluating such diagrams is time‐consuming, subjective, and difficult to scale, particularly when student solutions exhibit structural variations and diverse labeling choices. These challenges limit the consistency of grading and the timeliness of feedback provided to students. This article presents Automated Assessment of Use–Case Diagrams (AAUC), a feedback‐centric framework and tool for the automated assessment of UML use‐case diagrams, designed to support assessment practices in engineering education. The proposed approach integrates label and structure matching, along with instructor‐configurable marking rules, within a level‐aware assessment model that supports partial credit and differentiated evaluation. Unlike approaches that focus solely on correctness detection, AAUC emphasizes the generation of meaningful formative feedback by identifying specific modeling issues, such as missing actors, incorrect relationships, and inconsistent or incomplete labeling. The framework has been evaluated using a dataset of 445 student‐generated use‐case diagrams collected over multiple academic years from undergraduate software engineering and systems analysis courses. The evaluation demonstrates that the system can robustly assess diverse student submissions, apply consistent marking policies, and generate both quantitative scores and qualitative feedback aligned with instructor expectations. By combining automated assessment with explicit feedback generation, AAUC addresses scalability challenges while supporting iterative improvement in student modeling work. The framework provides a practical, extensible solution for instructors seeking to integrate automated support into engineering education contexts involving diagrammatic design tasks.
V. Vachharajani, J. Pareek· Computer Applications in Eng...· 0 citations
Background. Requirements inspection (RI) is a well-established practice for detecting potential defects in requirements artifacts early in the software lifecycle. Recent advances in large language models (LLMs) have stimulated interest in their potential to support requirements engineering (RE) tasks. However, empirical evidence on the effects of LLMs when used as collaborative assistants in human-performed RI remains scarce. Aims. We aim to investigate the impact of LLM support on human-performed RI, considering inspection effectiveness in terms of smell identification and severity classification (i.e., nocuous vs innocuous), as well as inspection duration. Method. We conducted a controlled crossover design experiment with 34 participants, who inspected textual specifications with and without LLM support, identifying and classifying requirements smells while recording inspection time. We analyzed the data using one Bayesian regression model per outcome variable, accounting for validity threats induced by the crossover design as well as covariates and mediators. Results. Results show that LLM support negatively affects smell detection accuracy but has no significant effect on smell classification or task duration. A learning effect is present across experimental periods, but reduced when RI is first performed with LLM support. Conclusions. Our findings provide empirical evidence that LLM support does not necessarily improve performance and may, instead, hinder it for novice inspectors. Moreover, the results suggest that learning RI with LLM-support from the beginning may slow down the skill acquisition process, implying threats for LLM-supported learning.
Giovanna Broccia, Julian Frattini, Chetan Arora et al.· 0 citations
