Background: Evaluation of LLM-based automated grading often relies on comparison with a single human score, which can obscure meaningful variability among raters of open-ended answers. This study introduces ConsensusGrade, a consensus-aware framework that treats the human reference as a scoring envelope rather than as a single point. Methods: We analyzed 1000 open-ended student answers from 100 students across 10 questions, each graded by four evaluators. Six previously generated and aligned automated grading configurations from GradeAgentOps were compared with the four-rater human reference. The score sets were generated using Llama 3.3 70B Instruct as the primary grader, with Qwen 2.5 14B Instruct for semantic repair. Results: Human evaluators showed meaningful agreement, with ICC(A,1) = 0.712, but exact four-rater agreement occurred in only 2.2% of records. Broad score dispersion occurred in 59.0%. All automated configurations showed negative bias relative to the human median. FULL achieved 68.5% inside-envelope positioning and a chance-adjusted score of 0.454; under the central-trimmed envelope, this rate decreased to 34.3%, while configuration ordering was preserved. Conclusions: ConsensusGrade provides a diagnostic framework for interpreting automated scores relative to observed human variability; inside-envelope rates should not be interpreted as stand-alone measures of grading accuracy.
Cătălin Anghel, A. Anghel, Mihai Vlase et al.· Applied System Innovation· 0 citations
Background: Large language models (LLMs) are increasingly explored for automated educational assessment, while future educational platforms may combine grading, feedback, learner analytics, and personalization. The objective of this study was to determine whether student-history metadata can influence the numerical score assigned to the same answer. Methods: This study introduces GradeDrift-LLM, a controlled framework for measuring student-history-induced score drift in LLM-based automated grading. We evaluated 1000 Computer Science answers from 100 students across six student-history conditions and eight open-weight LLMs. For each grading instance, the submitted answer, question, reference answer, rubric-related information, scoring scale, and grading instruction were kept constant; only the student-history condition varied. Results: Across 39,997 valid paired comparisons, 83.92% showed no drift, 9.40% showed upward drift, and 6.68% showed downward drift. Mean absolute drift was 0.2137 points, and the 95th percentile absolute drift was 1 point. Positive-history frames tended to increase scores, whereas negative-history frames tended to decrease them. Drift was model-dependent, not uniformly explained by approximate scale, and present in both technical and argumentative answers; rare extreme deviations reached 10 points. Conclusions: Student-history metadata can influence LLM-generated grading scores despite explicit instructions to ignore it. Future LLM-based grading systems should separate answer-based scoring from learner-context-based personalization and validate score invariance under controlled learner-context variations.
Cătălin Anghel, A. Anghel, M. Craciun et al.· Machine Learning and Knowled...· 0 citations