Bayan AlAmir

How consistent is the algorithm? Examining the intra- and inter-rater reliability of LLM-based writing assessment

Low consistency was observed for mechanics and grammatical accuracy, suggesting a need for clearer criteria or further model training, while high intra-rater reliability indicated high intra-rater reliability, particularly for task achievement and organization.