It is demonstrated that reliable AI-assisted assessment emerges from the integration of LLMs within structured and context-aware systems, rather than from model capabilities alone.
Abstract
Automated assessment of programming assignments remains a major challenge in higher education, particularly in large-scale courses where timely, consistent, and pedagogically meaningful feedback is difficult to provide, while Large Language Models (LLMs) have shown strong capabilities in code understanding and feedback generation, their use as standalone evaluators is fundamentally limited by inconsistency, lack of transparency, and weak alignment with instructional objectives. This paper argues that these limitations are not intrinsic to LLMs, but rather arise from their deployment as isolated components. In response, we propose a system-centric approach to AI-assisted assessment, introducing a hybrid framework that integrates LLMs within a structured, context-aware, and pedagogically aligned evaluation pipeline. The framework combines (i) explicit rubric-based decomposition of evaluation criteria, (ii) pedagogically guided prompting, and (iii) Retrieval-Augmented Generation (RAG) grounded in course-specific materials. Together, these components transform the evaluation process from a black-box prediction task into a traceable and reproducible decision process. The proposed approach is implemented in a real-world educational platform, EvaluaTeC, and evaluated on a dataset of 1287 programming submissions from 429 students. Experimental results show that the hybrid framework improves agreement with consolidated instructor reference grades (r=0.9059 vs. 0.7207 baseline), reduces evaluation error (MAE = 0.5134), and exhibited lower output variability in the recorded aggregate statistics, while maintaining practical latency and cost. Beyond numerical improvements, the system approximates key statistical properties of human grading and generates structured, pedagogically aligned feedback. These findings demonstrate that reliable AI-assisted assessment emerges from the integration of LLMs within structured and context-aware systems, rather than from model capabilities alone. This work contributes a principled framework for explainable and scalable automated assessment, advancing the design of trustworthy AI systems in education. This shift reframes automated assessment as a systems problem rather than a purely model-centric task.
An integrative synthesis design indicates that these controls can improve consistency, transparency, and pedagogical alignment of LLM-supported educational assessment while reducing, but not eliminating, hallucination, variability, bias, and over-reliance on model outputs.
Jussi S. Jauhiainen· Academia AI and Applications· 0 citations
Aim/Purpose: This systematic review synthesises 69 primary studies (2020–2025) to map how contemporary automatic question generation (AQG) research operationalises cognitive control, quality validation, and language inclusivity, and to identify where the field still needs to make progress.
Background: While large lang...
Firmansyah Ibrahim, Didik Dwi Prasetya, Triyanna Widiyaningtyas· Journal of Information Techn...· 0 citations
Large Language Models (LLMs) are increasingly used in educational contexts, yet their default answer-centric interaction paradigm conflicts with established principles of effective learning. Direct solution delivery can reduce productive struggle and encourage cognitive offloading rather than deep understanding. To add...
G. Kofler, Jamshid Mozafari, Adam Jatowt· 0 citations
Sustaining reading motivation in EFL and ESL settings remains a persistent pedagogical challenge. Textbook-based comprehension tasks—with predictable content, pre-printed answer keys, and limited interactivity—can produce passive compliance rather than engagement. The maturation of large-language-model (LLM) tools...
Peer code review is widely used in programming and software engineering education, but its instructional value depends on the quality of student feedback rather than review completion alone. In large classes, instructors can readily monitor participation but lack scalable methods for assessing whether feedback is relev...
Hsi-Min Chen, Hoang-Thanh Duong, Duong Thuy Trang Nguyen et al.· International journal of sof...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.