Can ChatGPT Rate Like a Human? A Many-Facet Rasch Analysis of Chinese EFL Argumentative Writing

Abstract

Assessing ChatGPT as an automated essay scoring tool requires measurement frameworks that go beyond correlations to diagnose the sources of scoring discrepancies. However, such studies remain scarce, and few have examined ChatGPT’s scoring behavior across rubric dimensions in the Chinese EFL context. The present study applied many-facet Rasch measurement together with the ESL Composition Profile to evaluate ChatGPT’s scoring of Chinese undergraduates’ EFL argumentative writing. Using a fully crossed design, five human raters and ChatGPT independently scored 366 essays. Three principal findings emerged. First, ChatGPT was the most severe rater, scoring approximately 10.67 points below the human raters’ average, with the discrepancy confirmed as systematic rather than random. Second, four of the six raters, namely ChatGPT and three of the five human raters, showed scoring consistency outside the range expected by the measurement model, with the largest departure observed for a human rater. Third, ChatGPT’s severity was not uniform across dimensions but followed a gradient from Content to Mechanics, indicating that its overall strictness was driven disproportionately by form-focused dimensions. Under the present uncalibrated zero-shot condition, ChatGPT functioned as a systematically distinct rater whose scores were not directly interchangeable with those of the calibrated human panel. Local calibration and validation would therefore be needed before considering operational use, while its dimension-level profile points to potential auxiliary roles in scoring workflows, such as first-pass screening of form-focused features.