ParaStudent, a fine-tuning framework for simulating novice programming revisions to support AI tutor evaluation, is introduced, demonstrating the promise of simulated engagement for pre-deployment feedback triage.
Abstract
Evaluating Artificial Intelligence (AI) tutor feedback before deployment requires anticipating student engagement, typically assessed through real interaction data. We introduce ParaStudent, a fine-tuning framework for simulating novice programming revisions to support AI tutor evaluation. Compared with prompted baselines, ParaStudent's revisions more closely match real student code distributions across functional, stylistic, and semantic metrics. Our best variant achieves AUCs of 0.80 for both feedback relevance and successful uptake when distinguishing streams with real engagement above versus at or below the median, while prompted baselines remain near chance on successful uptake. These findings demonstrate the promise of simulated engagement for pre-deployment feedback triage.
AI programming tutors provide scalable support, yet lack the behavioral context human tutors rely on to adapt support to learners'needs. We present TutorTrace, a dataset and behavioral abstraction pipeline that makes learners'behavioral context visible and computable in real time from low-level IDE telemetry. Across fo...
David Barron, Xiaohang Tang, Rezky Dwisantika et al.· 0 citations
Artificial intelligence offers an unprecedented opportunity to augment human capabilities, yet progress at the frontier has focused primarily on advancing model capabilities. We introduce StudentBench, a suite of AI teaching evaluations and a public platform that enables large-scale data collection with over 175,000 st...
Curtis G. Northcutt, Inaara Hasmani, Kevin Feng et al.· 0 citations
Fluent guidance is not the same as useful intervention. LLM tutors are typically trained to generate the next teacher utterance, implicitly assuming that every student turn warrants a response. However, our experiments indicate that this conflates tutoring capability (what to say) with intervention necessity (whether t...
Evaluation of five AI tutors from the Claude, Llama, Gemini, Qwen, and GPT families showed that relative rankings remained stable across learner states and interaction lengths, while absolute performance varied, revealing state-specific differences in tutor support.
This work presents StudentSim, a training framework that turns sparse per-student data into individualized simulators through pooled training followed by per-student specialization, and introduces StudentSimEval, a standardized protocol covering 60 students across chess, second-language English writing, and mathematics...
Ke Yang, Chenglong Wang, Michel Galley et al.· 1 citation
A five-level scale is developed, validated against human annotations, that characterizes responses according to the degree of direct assistance LLMs provide when students use them as tutors in authentic learning interactions and provides an empirical baseline for LLM assistance in tutoring interactions and a measuremen...
Suhyeon Lee, Juneha Baek, Jaehyeong Park et al.· 1 citation
