Probing, Not Tutoring: Design of a Conversational AI Agent for Introductory Mechanics and a Definition of "Cave Detection"

Abstract

Large language model (LLM) agents built for teaching tend to leak evaluation: they praise right answers, hint after wrong ones, and drift toward explaining. This note records the design of Talk It Through: Newton's Laws, a prompt-engineered agent for introductory mechanics that probes understanding instead of tutoring. In a short conversation, a student commits to a prediction about each of Newton's three laws, explains it in everyday words, and defends it against a changed case built from the student's own words that alters exactly one variable of the original scene. The agent never confirms, corrects, or hints during the conversation and scores only at the end. Each law receives one of four outcome codes: held, caved, self-corrected, or misconception survived. We define cave detection, the identification of a correct initial idea that is abandoned under a single changed case, as a measure of robustness that right/wrong item scoring cannot see. We summarize design rules found through more than 40 scripted simulation runs. All results reported here come from simulated prototype testing; no real-student data have yet been collected. A full design paper and an IRB-approved classroom study are in preparation. Preprint, not peer reviewed.