On the Non-Interchangeability of Deterministic and LLM-Based UML Evaluators

Abstract

Automated assessment of UML class diagrams is essential for scaling software engineering education, with educators adopting both deterministic evaluators and, in recent years, emerging solutions based on Large Language Models. Existing comparisons between the two approaches conflate differences in format and reasoning, making it difficult to identify the causes of disagreement and, consequently, to understand whether the two approaches can be used interchangeably. The goal of this study is to analyze the agreement between a fuzzy-matching deterministic evaluator and an evaluator based on Claude Sonnet (4.6 model) in a scenario where both evaluators consume the same format for diagrams and the same type of evaluation criteria and reference solution, and produce output in the same format, to identify whether reasoning differences affect score reliability and error sensitivity, while also discussing the implications related to deployment costs. We conducted our analysis by having the two evaluators assess 50 student diagrams taken from two different exam sessions from the Software Engineering course at Politecnico di Torino, measuring aggregate correlation related to diagram correctness, agreement across reported errors, and estimating LLM performance scaling in terms of time and cost. Results showed a strong aggregate correlation on the combined dataset; however, it also dropped substantially within each exercise, revealing that the overall correlation is inflated by the separation between exercise clusters rather than by within-exercise-agreement. Furthermore, agreement on error types showed divergences, showing that the deterministic evaluator flagged structural anti-patterns more frequently than the LLM-based one, which in turn flagged completeness violations more frequently. These findings, together with the cost and time analysis, point to the fact that deterministic and LLM evaluators are complementary rather than interchangeable, and that per-type agreement analysis should be performed to understand where each approach is preferred, and in which cases one is more suited than the other.