Mapping the Distribution and Depth of Metacognitive Processes in Generative AI-Assisted Learning: Evidence from Interaction Logs and Concurrent Think-Aloud Protocols
TL;DR
Combining the sources provides a fuller account of metacognitive activity than either record alone, without establishing monitoring accuracy, process effectiveness, or learning outcomes.
Abstract
Generative artificial intelligence (GenAI) is increasingly used as a cognitive aid in education, yet no single process measure captures how learners monitor and control GenAI-assisted work. Guided by a reciprocal monitoring-control account of metacognition, this descriptive, exploratory study examined what interaction logs and concurrent think-aloud speech make observable during a 90 min instructional-design task. Seventy-nine undergraduates from 11 majors completed the task with DeepSeek support. The same two-dimensional rubric was applied separately to both records: each retained semantic unit received one primary operational category (planning, monitoring, or regulation) and one evidence-depth label (implicit, awareness, strategic, or reflective). Planning was treated as prospective control and regulation as adaptive control after evaluation or feedback. The interaction-log and think-aloud records yielded 274 and 193 retained instances, respectively. In both sources, regulation was the most frequent primary label and monitoring the least frequent, while evidence-depth ratings were concentrated at awareness and strategic. The combined strategic and reflective share was 25.4 percentage points higher in think-aloud than in interaction-log monitoring units and 9.9 points higher in think-aloud than in interaction-log planning units; for regulation, it was 14.0 points higher in interaction-log than in think-aloud units. These aggregate contrasts are descriptive because coded instances are nested within participants and the sources are paired within individuals. The central contribution is that verbalized monitoring information and enacted control leave different observable traces. Combining the sources therefore provides a fuller account of metacognitive activity than either record alone, without establishing monitoring accuracy, process effectiveness, or learning outcomes.