Large language models as grading assistants in public health education: a method-comparison study of essay-style exam assessment
TL;DR
Some LLMs, particularly ChatGPT and Kimi, showed moderate alignment with human examiners, but the alignemnet was not consistently stable across repeated sessions or operational settings.
Abstract
Large language models (LLMs) are increasingly being considered for assessment support in health professions education; however, evidence of their performance in essay-style examinations remains limited. In particular, little is known about the reproducibility and operational stability of LLM-based grading under different conditions. This study compared the grading performance of several LLMs with that of human examiners in a master's-level public health course and assessed the consistency of LLM-based grading across repeated sessions and different file upload volumes. We conducted a method comparison and validation study using anonymized student submissions from a 5-hour essay-style examination in a master's-level course in public health, empowerment, and health promotion. Four LLMs—ChatGPT, Gemini, LeChat, and Kimi—were prompted to assign final grades on an A-F scale using the same grading guidance as human examiners. The agreement between the LLM- and human-assigned grades in a single-file upload setting was assessed using weighted Cohen's kappa. Internal consistency across file upload volumes was assessed using Krippendorff alpha. Repeated grading across multiple sessions was performed for the best-aligned model. Agreement with human examiners was limited in fast mode and improved in reasoning or thinking modes; however, reproducibility across sessions and implementation conditions remained limited. In the single-file upload setting, ChatGPT showed the strongest agreement with human examiners (weighted kappa 0.718, 95% CI 0.578–0.859), followed by Kimi (weighted kappa 0.571, 95% CI 0.373–0.768). ChatGPT achieved 50.0% exact agreement and 90.6% agreement within ±1 grade, with corresponding values of 28.1% and 78.1% for Kimi. Gemini and Gemini Pro showed the highest internal consistency across the file upload conditions. Repeated grading by ChatGPT across 5 days showed moderate variation. LLMs used fewer extreme grades than did human examiners. Some LLMs, particularly ChatGPT and Kimi, showed moderate alignment with human examiners, but the alignemnet was not consistently stable across repeated sessions or operational settings. LLMs may currently be better suited as supervised grading assistants than as autonomous graders.
