AI-Based Assessment of Revision-Associated Research Writing Development in a Peer-Feedback-Supported
TL;DR
Artificial intelligence-supported analysis identified writing development associated with revision and captured human judgment to some extent, but did not provide causal evidence of the independent contribution of peer feedback to writing improvement due to the observational design.
Abstract
Despite its ubiquity as a writerly practice in academic writing, there is little publicly available evidence that connects feedback-based revision practices with quantitative changes in research writing. This study proposes a framework based on artificial intelligence (AI) that aids in the assessment of revision-related development in student research exposés based on the Exposía academic writing and peer-feedback corpus (Ec). The raw files analyzed included 3247 records, with 16,068 feedback and comment records, 55 matched draft–final pairs of exposés, and 28 matched draft–final pairs of cases by score. For every text, features of NLP were computed, such as: word count, type–token ratio, moving-average type–token ratio over 100-word windows (MATTR-100), lexical density, readability, and adjacent-sentence cohesion. The differences between the draft and the final text were analyzed with paired statistical tests, and the human assessment scores were estimated using draft-Ridge Regression and draft-Random Forest models. Human assessment scores increased significantly from the draft (M = 5.26) to the final version (M = 7.55), t(27) = 9.04, p < 0.001, Cohen’s dz = 1.709. There was also a significant increase in word count, MATTR-100, and lexical density. Random Forest gave a better R2 (0.426) than Ridge Regression (R2 = 0.248). The exploratory feedback-improvement model resulted in a negative R2 due to a limited number of automatically merged feedback features, a lack of alignment between comment and revision, and a small number of matched scores. Therefore, AI-supported analysis identified writing development associated with revision and captured human judgment to some extent, but did not provide causal evidence of the independent contribution of peer feedback to writing improvement due to the observational design. The framework offers a repeatable benchmark and calls for enhanced detail in providing feedback–revision alignment.