Embodied multimodal human-computer interaction for English language learning: integrating speech, gestures, emotion, and adaptive feedback

Abstract

Current English language learning systems are often limited by single-dimensional learning-state perception, insufficient modeling of affective-cognitive characteristics, and delayed or inflexible feedback regulation. To address these limitations, this study proposes an affective-cognitive gated cross-modal transformer (ACG-CMT) for embodied multimodal human-computer interaction in English language learning. The framework integrates learners’ speech, gestures, and affective responses as multimodal inputs. The speech encoder learns low-level acoustic-prosodic representations from Mel-spectral, energy, pitch/prosodic, and temporal patterns rather than receiving rubric-derived target scores; a spatiotemporal motion encoder captures gesture and posture dynamics; and an affective encoder models facial-action and vocal-affect cues. A cross-modal transformer captures temporal dependencies and contextual interactions among linguistic output, embodied behavior, and affective fluctuations. During fusion, an affective-cognitive gate adjusts modality-specific contributions using latent affective/cognitive context and pre-decision task context available at inference time, without access to ground-truth target labels. A multi-task decision module then supports learning-state recognition, affective-cognitive load estimation, and feedback-policy matching. ACG-CMT achieves an accuracy of 0.892, a macro-F1 of 0.884, an AUC of 0.941, and a difficult-state F1 of 0.877 for learning-state recognition; R 2  = 0.836, RMSE = 0.084, and MAE = 0.068 for the operational affective-cognitive load reference index; and a feedback-policy matching rate of 0.861 with an average latency of 438 ms. These results indicate that the framework can integrate multimodal behavioral information for learner-state estimation and adaptive feedback, while the internal gating patterns are interpreted as model-level associations rather than causal psychological mechanisms.