Skip to content
Book Open access

Multimodal Learner-State Analytics: Cross-Modal Insights from Motor-expressions and EEG for Instructor-Facing Visualization Platform

Oct 2026 · Proceedings of the 28th International Conference On Multimodal Interaction · 0 citations · 75 references

Abstract

Remote video lectures expand educational access but limit instructors’ ability to observe how students engage with instructional content. To address this challenge, we explore how insights from portable electroencephalography can enrich visual analytics for computer-based learning environments. In our study, sixteen middle school students watched a mathematics video lecture while we collected and temporally aligned their gaze data, video recordings of motor expressions, and physiological indices of attention, valence, and arousal. By comparing these measures, we developed models to estimate internal learning states from accessible motor-expression data alone. We then applied changepoint detection to identify critical learning moments and integrated these alongside the estimated trends into a visualization platform designed for educators. An evaluation with thirty teachers revealed that observing these critical moments helped them identify specific lecture segments warranting closer inspection, allowing educators to reflect deeply on pacing, explanation quality, and overall slide design. These findings highlight a practical path toward scalable visualization platforms for digital education, equipping educators with profound pedagogical insights through accessible sensing alone.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.