Jul 2026· Annual Conference on Innovation and Technology in Computer Science Education· 0 citations· 18 references
Computer Science
TL;DR
Socratic AI, a VS Code-integrated tutor that addresses this through pedagogically-grounded Socratic dialogue constrained to withhold direct solutions is presented, and evidence that stateful tracking enables adaptive Socratic dialogue that scaffolds productive struggle rather than short-circuiting learning is contributed.
Abstract
Large Language Models in programming education face a fundamental tension: while capable of generating correct solutions instantly, doing so undermines the struggle necessary for conceptual learning. We present Socratic AI, a VS Code-integrated tutor that addresses this through pedagogically-grounded Socratic dialogue constrained to withhold direct solutions. Our core innovation is a stateful misconception detection mechanism that tracks student understanding across dialogue turns. Operating on a taxonomy of foundational programming errors (off-by-one, mutation vs. reassignment, return vs. print, async/parallel confusion), the system evaluates each student utterance to produce per-misconception verdicts (reinforced, weakened, introduced, absent) with certainty scores. These accumulate through small, incremental confidence score updates until a derived resolution threshold is achieved. This enables the tutor to persist on specific conceptual gaps rather than abandoning them when students deflect or express frustration. This stateful tracking enables two key capabilities: (i) the system adapts questioning intensity based on detected learner confidence signals, shifting from diagnostic probes to conceptual contrasts, and (ii) enforces purely Socratic constraints through multi-layered validation that blocks code generation, direct explanations, and multi-question responses. The system is implemented as a VS Code extension with Gemini-based classification back-end and cloud database persistence for longitudinal analysis. In a preliminary deployment with 48 students, we observed increased question specificity, frequent learner articulation consistent with mental model revision, and reduced recurrence of targeted errors on similar tasks. We contribute a deployable architecture, a misconception taxonomy, and evidence that stateful tracking enables adaptive Socratic dialogue that scaffolds productive struggle rather than short-circuiting learning.
Generative AI tutors have become a common tool for independent learning, yet their capacity to support self-regulated learning (SRL) is poorly understood. This simulation-based textual analysis of prompt design evaluates a frontier large language model (Claude Sonnet 4.6) as a tutor across 60 scripted sessions on a single topic (density), crossing three levels of SRL-informed system prompting (Minimal, Moderate, Extensive) with four learner-behavior variants (Standard, Misconception, Disengagement, Overconfidence). Tutoring transcripts were scored on a 14-dimension framework spanning SRL phases, SRL developmental stages, self-determination theory principles, and Merrill’s First Principles of Instruction, applied via an LLM judge. Adding SRL context to the system prompt raised total tutoring scores, but only at the Extensive SRL support level. Minimal and Moderate prompting produced the same performance, near 36 on a 70-point scale, and Extensive prompting raised it to 40, a statistically significant effect (partial η2 = 0.24). The learner’s behavior in the session had a larger effect than the prompt did (partial η2 = 0.37), with disengaged learners scoring lowest. The threshold pattern held under an independent judge from a different developer than the tutor model. The findings support a method for evaluating GenAI tutors empirically and point to dynamic, dialogue-aware prompting alongside explicit SRL scaffolding.
Kendall Hartley, Fabiola Sáez-Delgado, Javier Mella-Norambuena· Future Internet· 0 citations
SocraticTrap-CS is introduced, a publicly available benchmark that probes the capacity of open-weight LLMs to generate strategic misconceptions on demand and reframes the evaluation of educational LLMs around pedagogical trustworthiness rather than factual correctness alone.
Marijela Miličević, Mia Rovis, Ratomir Karlović et al.· Information· 0 citations
Large language models (LLMs) deployed in educational settings often behave as direct answerers: they disclose target concepts in the opening turn instead of guiding students through progressive inquiry, as Socratic pedagogy prescribes. We present HeuristicEdu, a two-phase pipeline that aligns Qwen2.5-7B toward Socratic tutoring via supervised warm-up and Group Relative Policy Optimization (GRPO). Training uses SocraticEdu, 797 multi-turn Chinese children's science dialogues reconstructed from a live platform, with a heuristic reward over cognitive depth (R_cog), curiosity engagement (R_eng), and directness (R_dir), together with a K_query correction for student-introduced terms. We introduce Scaffolding Effectiveness (SE) and Conversation Depth (CD) to evaluate outcomes beyond surface fluency. On 30 held-out questions, the best GRPO variant improves SE from 30.0% to 63.3% and lowers keyword leakage from 30.0% to 13.3%. Notably, this best variant omits the directness penalty during optimization, suggesting that explicit anti-leakage terms can conflict with gradient-based behavioral alignment. An unaligned Qwen-72B baseline reaches 0% SE and 96.7% leakage, showing that scale alone does not induce Socratic behavior.
Xiaokun Wang, Siyu Song, Wentao Liu et al.· 0 citations
Mastering loops and repetition is a well-documented source of difficulty for novice programmers in upper primary school, who often “unroll” a repeated action into a long linear stack of identical blocks rather than recognising the repeating pattern and expressing it with a single repeat block. This study reports a between-groups experiment. We examined whether a multimodal artificial intelligence (AI) assistant tutor — one that interprets screenshots of pupils’ Scratch code and replies with Socratic scaffolding rather than with corrections — improves learning outcomes and engagement relative to traditional classroom lesson. Forty pupils aged 11 to 12 were assigned to two intact classes of 20. The first received a standard, teacher-led lesson on loops, in pairs at shared computers using scratch.mit.edu. The second covered the same material; whenever a pair’s script did not behave as intended, the pair captured a screenshot (a “print screen”) of their blocks and submitted it to the multimodal AI assistant tutor, which read the code and replied with a guiding question that prompted the pair to locate and repair the fault themselves. Pairs followed the driver–navigator pair-programming technique on both groups. We measured learning with a matched pre- and post-test on loops and gauged engagement through a short attitude questionnaire and classroom observation. The tutor group scored significantly higher on the post-test and reported markedly greater interest and enjoyment than the traditionally taught group. These findings suggest that multimodal AI tools that read learner code and scaffold through questioning can enhance both achievement and motivation in introductory programming.
Dimitrios Kravvaris· European Journal of Contempo...· 0 citations
Providing high-quality feedback on student work is essential for learning, yet delivering such feedback at scale remains challenging. In this paper, we focus on feedback for open-ended short answer questions in introductory programming, with the goal of nudging students toward success on reattempts without revealing the correct answer. We develop a five-criteria rubric grounded in educational literature for evaluating feedback quality: (1) acknowledging correct portions of the student answer, (2) identifying at least one flaw (if present), (3) providing actionable guidance for improvement, (4) maintaining appropriate concealment of the answer, and (5) using an appropriate conversational tone. Using this rubric, we compare feedback generated by a frontier LLM (OpenAI o1) to feedback from nine teaching assistants across 90 student responses, with three researchers and an LLM independently scoring all feedback. Our results show that while one TA often produced the best feedback, the LLM demonstrated consistently higher average performance than TAs, as evaluated by humans. However, we also uncover significant self-preference bias when using LLMs to evaluate feedback quality: the LLM systematically rated its own outputs higher than human experts did. This bias, which research suggests persists even in cross-model evaluation, raises important methodological concerns for researchers employing LLM-based evaluation. We provide detailed characterization of both TA and LLM performance, analyze sources of variance in TA feedback quality, and discuss implications for deploying LLM-generated feedback in educational settings.
Binglin Chen, Rajarshi Haldar, Max Fowler et al.· 0 citations
Large language models (LLMs) often exhibit sycophancy, optimizing for agreement over productive challenge, which severely limits their utility in domains like professional skills training, where growth requires pushback. We introduce, ConvoDojo, a novel conversational AI platform for practicing difficult workplace conversations, engineered not merely as a commercial training application but also as a flexible, instrumented research platform for evaluating conversational AI strategies. ConvoDojo repurposes LLMs as structured sparring partners to support skill development in difficult workplace conversations (e.g., performance feedback, conflict resolution), addressing the reported managerial tendency to avoid them. This paper showcases the platform and presents an evaluation of how key conversational user interface (CUI) design elements, namely, the addition of structured feedback and upfront instructional scaffolding, impact managers’ learning. Results show that ConvoDojo is highly engaging and promotes user reflection. We demonstrate how theory-informed dialogue and adaptive pushback can transform an LLM into an effective, measurable tool for complex communication skills development.
Everlyne Kimani Cross, Luiza A Santos, Laurent Denoue et al.· International Conference on...· 0 citations
