Can AI grade like a nurse educator? Benchmarking large language models for automated scoring of open-ended assessments in nursing education: A descriptive cross-sectional study.
Abstract
Background
Open-ended, case-based assessments are essential for evaluating clinical reasoning in nursing education, yet their high-scoring burden limits their widespread use, particularly in large cohorts. Large language models (LLMs) may reduce this barrier, but empirical evidence in nursing education remains scarce.
Objectives
To examine the agreement between rubric-guided LLM scoring and academic consensus, evaluate the consistency of LLM scoring across repeated runs, and explore implications for educator workload and assessment scalability in undergraduate women's health nursing education.
Design
A descriptive, cross-sectional study framed within programmatic assessment theory.
Participants
215 undergraduate nursing students enrolled in a women's health nursing course at a Turkish university during the 2025-2026 academic year.
Methods
Students completed a case-based assessment comprising five clinical vignettes with 15 open-ended questions. Three faculty members independently scored a benchmark subset (22 scripts; 330 responses) using an analytic rubric. Two cost-effective LLMs (GPT-4.1 mini, Gemini Flash 3 Preview) scored the same subset across four repeated runs. Agreement was assessed using intraclass correlation coefficients (ICC), mean bias, mean absolute error, and Pearson correlations. Run-to-run stability was evaluated across the four scoring iterations.
Results
Inter-rater reliability among academics was excellent (ICC = 0.97). GPT-4.1 mini demonstrated strong agreement with academic consensus (ICC (2,1) = 0.930; bias = -3.06 points; Pearson r = 0.946), with 90.9% of scores falling within ±10 points. Gemini Flash 3 Preview showed a systematic leniency bias (+16.57 points) despite strong ranking ability (r = 0.917). Both models exhibited high run-to-run stability (ICC (2,4) = 0.990 and 0.992). Faculty reviewers confirmed that automated feedback was rubric-aligned and actionable.
Conclusions
Rubric-guided LLM scoring can achieve close agreement with academic markers for open-ended nursing assessments, though model selection and local calibration are critical. When integrated with human oversight, this approach can support scalable, feedback-rich assessment practices consistent with programmatic assessment principles.