Xiaoliang Zhou

Evaluating AI-Based Automated Essay Scoring Through Signal Detection Theory: Beyond Aggregate Agreement Metrics.

Signal Detection Theory is applied to evaluate eight state-of-the-art LLMs to provide a robust framework for calibrating and selecting AI scoring systems based on specific pedagogical goals and fairness requirements and demonstrates that low human-machine agreement stems from both a deficit in discriminating accuracy a...