Skip to content
Open access

From Black-Box Grading to Pedagogically Aligned AI Assessment: A Hybrid LLM–RAG Framework for Explainable and Scalable Automated Code Evaluation

Sep 2026 · Applied Sciences · Vol 16, pp. 9268 · 0 citations · 35 references

TL;DR

It is demonstrated that reliable AI-assisted assessment emerges from the integration of LLMs within structured and context-aware systems, rather than from model capabilities alone.

Abstract

Automated assessment of programming assignments remains a major challenge in higher education, particularly in large-scale courses where timely, consistent, and pedagogically meaningful feedback is difficult to provide, while Large Language Models (LLMs) have shown strong capabilities in code understanding and feedback generation, their use as standalone evaluators is fundamentally limited by inconsistency, lack of transparency, and weak alignment with instructional objectives. This paper argues that these limitations are not intrinsic to LLMs, but rather arise from their deployment as isolated components. In response, we propose a system-centric approach to AI-assisted assessment, introducing a hybrid framework that integrates LLMs within a structured, context-aware, and pedagogically aligned evaluation pipeline. The framework combines (i) explicit rubric-based decomposition of evaluation criteria, (ii) pedagogically guided prompting, and (iii) Retrieval-Augmented Generation (RAG) grounded in course-specific materials. Together, these components transform the evaluation process from a black-box prediction task into a traceable and reproducible decision process. The proposed approach is implemented in a real-world educational platform, EvaluaTeC, and evaluated on a dataset of 1287 programming submissions from 429 students. Experimental results show that the hybrid framework improves agreement with consolidated instructor reference grades (r=0.9059 vs. 0.7207 baseline), reduces evaluation error (MAE = 0.5134), and exhibited lower output variability in the recorded aggregate statistics, while maintaining practical latency and cost. Beyond numerical improvements, the system approximates key statistical properties of human grading and generates structured, pedagogically aligned feedback. These findings demonstrate that reliable AI-assisted assessment emerges from the integration of LLMs within structured and context-aware systems, rather than from model capabilities alone. This work contributes a principled framework for explainable and scalable automated assessment, advancing the design of trustworthy AI systems in education. This shift reframes automated assessment as a systems problem rather than a purely model-centric task.

Read PDF

Similar papers

Review Open access Sep 2026

Generative AI and LLMs in educational assessment: challenges, evidence, and a process-based framework

An integrative synthesis design indicates that these controls can improve consistency, transparency, and pedagogical alignment of LLM-supported educational assessment while reducing, but not eliminating, hallucination, variability, bias, and over-reliance on model outputs.

Jussi S. Jauhiainen · 0 citations
Review Open access 2026

Beyond Fluency: Cognitive Control and Pedagogical Quality in LLM-Based Automatic Question Generation — A Systematic Review of Bloom-Aligned Assessment Practices (2020–2025)

Aim/Purpose: This systematic review synthesises 69 primary studies (2020–2025) to map how contemporary automatic question generation (AQG) research operationalises cognitive control, quality validation, and language inclusivity, and to identify where the field still needs to make progress. Background: While large lang...

Firmansyah Ibrahim, Didik Dwi Prasetya, Triyanna Widiyaningtyas · 0 citations
Book Open access Nov 2026

HintLab: An Interactive Tool for Hint Generation, Analysis and Evaluation

Large Language Models (LLMs) are increasingly used in educational contexts, yet their default answer-centric interaction paradigm conflicts with established principles of effective learning. Direct solution delivery can reduce productive struggle and encourage cognitive offloading rather than deep understanding. To add...

G. Kofler, Jamshid Mozafari, Adam Jatowt · 0 citations
Open access Oct 2026

Can AI make reading more engaging? A motivational framework for AI-assisted reading assessment in EFL/ESL classrooms: a practitioner case study

Sustaining reading motivation in EFL and ESL settings remains a persistent pedagogical challenge. Textbook-based comprehension tasks—with predictable content, pre-printed answer keys, and limited interactivity—can produce passive compliance rather than engagement. The maturation of large-language-model (LLM) tools...

M. Almirabi · 0 citations
Review Oct 2026

Making Feedback Quality Visible to Instructors: An AI-Supported Learning Analytics Dashboard Linking Peer Code Review to Pedagogical Action in Programming Education

Peer code review is widely used in programming and software engineering education, but its instructional value depends on the quality of student feedback rather than review completion alone. In large classes, instructors can readily monitor participation but lack scalable methods for assessing whether feedback is relev...

Hsi-Min Chen, Hoang-Thanh Duong, Duong Thuy Trang Nguyen et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.