FATE (FLC AI Tutor Evaluator), a specialized 8B-parameter language model designed to evaluate AI tutors, is introduced, which assesses pedagogical ability across Mistake Identification, Mistake Location, Guidance, and Actionability.
Abstract
The rapid integration of Large Language Models (LLMs) into K-12 and higher education has outpaced the development of reliable methods for evaluating their pedagogical quality. As the research community starts to explore the space of automating evaluation of AI tutors, we introduce FATE (FLC AI Tutor Evaluator), a specialized 8B-parameter language model designed to evaluate AI tutors. Aligned with the four core evaluation tracks from the BEA 2025 Shared Task, our model assesses pedagogical ability across Mistake Identification, Mistake Location, Guidance, and Actionability. Because pedagogical evaluation is a specialized task with limited labeled data, we leverage knowledge distillation from a frontier LLM to generate additional supervision, yielding absolute performance gains up to 22.63 percentage points. Finally, we demonstrate FATE's utility as an automated evaluator by benchmarking instructional responses generated by popular commercial models, including ChatGPT, Claude, Gemini, and DeepSeek. On average, we have found that Gemini 2.5 Flash perfomed best (82.88%), then ChatGPT 5.5 Instant (80.75%), followed by DeepSeek V4 Flash (80.13%) and Claude Sonnet 4.6 (74.00%).
CSTutorBench is introduced, a benchmark for evaluating language models as CS tutors in VEX VR, a block-based robotics environment, and preliminary findings reveal that models perform well on surface-level criteria such as vocabulary and tone but struggle with deeper pedagogical behaviors.
Generative AI tutors have become a common tool for independent learning, yet their capacity to support self-regulated learning (SRL) is poorly understood. This simulation-based textual analysis of prompt design evaluates a frontier large language model (Claude Sonnet 4.6) as a tutor across 60 scripted sessions on a single topic (density), crossing three levels of SRL-informed system prompting (Minimal, Moderate, Extensive) with four learner-behavior variants (Standard, Misconception, Disengagement, Overconfidence). Tutoring transcripts were scored on a 14-dimension framework spanning SRL phases, SRL developmental stages, self-determination theory principles, and Merrill’s First Principles of Instruction, applied via an LLM judge. Adding SRL context to the system prompt raised total tutoring scores, but only at the Extensive SRL support level. Minimal and Moderate prompting produced the same performance, near 36 on a 70-point scale, and Extensive prompting raised it to 40, a statistically significant effect (partial η2 = 0.24). The learner’s behavior in the session had a larger effect than the prompt did (partial η2 = 0.37), with disengaged learners scoring lowest. The threshold pattern held under an independent judge from a different developer than the tutor model. The findings support a method for evaluating GenAI tutors empirically and point to dynamic, dialogue-aware prompting alongside explicit SRL scaffolding.
Kendall Hartley, Fabiola Sáez-Delgado, Javier Mella-Norambuena· Future Internet· 0 citations
EduMind is introduced, a unified tutoring and assessment platform designed around a dual-track evaluation model that demonstrates how assessment and tutoring can be unified into a seamless workflow, and remained operationally stable throughout all testing phases.
Dhyan Gowda, M. G. Aruna, Priya Ramesh Prasad et al.· International Journal of Sci...· 0 citations
This article presents a novel approach to Intelligent Tutoring Systems (ITS) by integrating Retrieval-Augmented Generation (RAG) with Large Language Models (LLMs) to enable dynamic personalization in educational contexts. The system addresses limitations in traditional ITS that rely on static, rule-based approaches by implementing a three-layered architecture combining semantic retrieval mechanisms with generative AI capabilities. Using GPT-4 as the core LLM enhanced with a custom RAG framework, the system demonstrates improvements in response accuracy (93%), inference speed (2.1 seconds per prompt), and computational efficiency compared to a standard GPT-4 baseline, a traditional rule-based ITS, and an LLM with keyword-based retrieval. The research employs both ASSISTments (fine-grained interaction data) and EdNet (large-scale longitudinal data) datasets for evaluation. Results show that the RAG-enhanced system achieves 40% better contextual relevance compared to standard LLM implementations. The framework incorporates adaptive prompting strategies, real-time knowledge base updates, and multi-level personalization algorithms to create a dynamic educational environment.
Kuyoro Afolashade, N. Uchenna, Akinwunmi Damilare· British journal of computer,...· 0 citations
Agentic tutoring systems increasingly rely on large language models (LLMs) to generate real-time feedback, yet risky outputs, such as answer leakage, contradiction, and hallucination, can directly affect learning. Answer leakage is a central concern in this setting. It exposes problem solutions to students and can be verified through human annotation. This article presents a pedagogical style-conditioned audit protocol for automated evaluators that detect answer leakage in LLM-generated tutoring feedback before deployment. Using 1100 mathematics problems from the MATH dataset and 4400 feedback items generated under Baseline, Direct, Socratic, and Encouragement styles, we compare rule-based, embedding-based, natural language inference-based, and two LLM-as-a-Judge evaluators from different model families (Qwen2.5-32B and Gemma3-27B) and validate them against a 314-item human-annotated gold standard. The results show that pedagogical style changes how answer-leakage signals appear in feedback and how reliably automated evaluators detect them. Rule-based and LLM-Judge evaluators that perform well under Baseline, Direct, and Encouragement show consistent degradation under Socratic feedback. The primary LLM-Judge (Qwen2.5-32B) achieves overall F1 = 0.726 but drops to 0.427 on Socratic items, and a second LLM-Judge (Gemma3-27B) reproduces this degradation with Socratic F1 = 0.342. The four factorial prompt-sensitivity variants reach overall F1 up to 0.844, but their Socratic F1 remains at or below 0.390. These findings show that automated tutoring-feedback auditing must be style-specific, prompt-calibrated, and human-grounded before deployment.
Wenzhuo Song, Xuechen Wang, Shuang Liang et al.· IEEE Transactions on Learnin...· 0 citations
Large language models (LLMs) are increasingly used as AI tutors, but a correct answer is not always a pedagogically appropriate one. In classroom learning, effective help depends not only on correctness, but also on whether a response matches the learner's current foundation, the course sequence, and the timing of concept introduction. Existing evaluations focus mainly on answer quality, leaving this instructional fit under-measured. We present the Pedagogical Suitability Index (PSI), a composite metric of six theory-informed sub-scores that evaluates how well LLM-generated tutoring responses align with learner readiness and curricular progression, and we further use PSI as a structured feedback signal for response improvement. We evaluate four LLM tutors (ChatGPT, Gemini, Gemma4, and Qwen3) across 240 scenario-based evaluations using paired standard and defective prompts, then apply a PSI-guided regeneration protocol to 62 weak-performing cases. Baseline differences across the four tested models were modest overall (PSI range: 0.557 to 0.638), and open-weight and closed models did not exhibit a clear separation in pedagogical fit. Under the tested prompt perturbations, overall PSI remained largely stable (Delta = -0.002), though sub-score trade-offs emerged. More importantly, PSI-guided feedback substantially improved weak-performing cases: 51 of 62 cases improved (82.3%). Focused manual evaluation of the 62 PSI-selected weak cases provides initial evidence that the identified weaknesses are instructionally meaningful and that many PSI-guided regenerations correspond to human-judged improvement. These results suggest that learner- and curriculum-aware alignment may matter more for effective tutoring than model category alone, and that such alignment is both measurable and improvable.
Benjamin Barlog, Hudson Craig, Zedong Peng· 0 citations