Investigating LLM–Human Rater Agreement in Automated Essay Scoring of Turkish L1 EFL Learners

Abstract

This study examined how closely a large language model (LLM) agreed with expert human raters when scoring English argumentative essays by Turkish L1 university preparatory students, and whether human rater and LLM scores distinguished two proficiency groups. A total of 110 handwritten essays from an elementary and a pre-intermediate group were scored by two expert raters and by an LLM applying a single zero-shot prompt built around a holistic rubric derived from the program's criteria. Agreement and association were estimated with quadratic weighted kappa and correlation measures, and group differences with Mann-Whitney U tests. Agreement between the LLM and each human rater was moderate and comparable to the agreement between the raters themselves, whose reliability was itself only moderate (ICC(2,1) = .472). A direct comparison found no significant difference between human–human and human–LLM agreement. It was weaker in the elementary group, and all three sources ranked the pre-intermediate group higher. The results point to a complementary use of LLM-based scoring alongside human judgment rather than a replacement for it.