Understanding Student Effectiveness with AI Chatbots in Introductory Programming Education: An Interaction Analysis Study

Abstract

Generative AI chatbots are increasingly used in introductory programming courses, but whether they support learning or encourage over-reliance remains unclear. To examine which interaction features are associated with productive persistence, we analyzed 198 CS1 student–chatbot conversation logs using a five-dimensional coding scheme: query type, response relevance, response type, dialogue outcome, and effectiveness marker. Persistence was measured as the number of student prompts per log. Negative binomial models showed that relevance failures predicted longer interactions, particularly false positives (IRR = 2.49) and false negatives (IRR = 1.65), both p < 0.001. Logistic models showed that false negatives strongly predicted unresolved sessions (OR = 9.94, p = 0.008), suggesting that the inability to answer accelerates abandonment. In contrast, conceptual (Socratic) responses predicted progressive effectiveness markers. Overall, interaction quality, not mere usage, appears to drive productive persistence, highlighting the importance of minimizing false negatives and encouraging Socratic scaffolding in educational chatbot design and deployment.