Designing for Self-Regulated Learning in AI-Assisted Quizzing: A Classroom Pilot Study

Abstract

AI support is increasingly embedded in online quizzes, yet instructors often lack clear, actionable signals about where students struggle during those assessments. We present a classroom-deployed quiz system that combines integrity-preserving hinting (TA-AI), trace summarization (Analytics-AI), and feedback-driven refinement (AI-Improver) to generate instructor diagnostics from routine interaction logs. The system was used in a graduate assembly programming course over five quiz weeks (N = 18). We report deployment evidence focused on reliability and instructional usefulness for monitoring: promptintent coding reached substantial agreement (Cohen’s kappa [κ] = 0.81); fixed-effects models (with student and item controls) showed a negative association for one-hint interactions (odds ratio [OR] = 0.231, indicating approximately 77% lower odds of a correct response for single-hint interactions relative to 0-hint interactions); and item-level demand spikes were operationalized via a demand × success prioritization process for weekly review. Rather than producing automated judgments or claims of causal learning gains, the analytics are designed as practical prioritization cues that direct instructor attention toward high-need items during AI-assisted quizzes.