Leveraging Multimodal Data to Understand Computational Thinking in Young Learners: Augmented Reality and Social Robots

Abstract

This research examines the computational thinking abilities of young children (2nd-year elementary students) within an embodied learning framework aimed at enhancing STEM education. The study investigates how augmented reality and social robots can support problem-solving in STEM-related tasks. Over four days, students engaged with a specially designed educational tool that encouraged physical interaction and social engagement with a robot. The interactions were tracked to monitor changes in their movement, problem-solving behavior, and robot engagement. To analyze learning behaviors, the research team manually annotated video recordings, focusing on key indicators of engagement and cognition. Simultaneously, data from motion capture systems and facial muscle sensors were collected to provide objective measurements of physical responses. A statistical analysis will identify patterns between the sensor data and human annotations. Subsequently, a machine learning model will be developed to link sensor data with behavioral annotations, enabling automatic recognition of learning behaviors in future studies. This model aims to streamline the assessment of children's interactions in tech-enhanced learning environments. The study will also explore essential measures that reflect how children approach and adjust to STEM challenges, offering insights into the cognitive and physical aspects of embodied learning. This work has the potential to advance evaluation methods in STEM education, leading to improved educational tools and strategies for engaging young learners in computational thinking.