When Modalities Disagree: Multimodal Sentiment Conflict as an Affective Signal in Collaborative Learning

Abstract

Multimodal emotion recognition (MER) systems typically combine facial, vocal, and linguistic cues to arrive at a single consensus label, often treating any disagreement between modalities as noise. However, we argue that in collaborative learning settings, such disagreements can provide valuable emotional signals. We present a framework that detects and characterizes mismatches in sentiment polarity—positive, neutral, and negative—across facial expressions, speech tone, and transcribed speech, without fusing the different modalities. Using valence-arousal estimation for facial and vocal expressions, as well as sentiment classification for text, we analyzed 1,342 utterance-aligned segments from student dyads in a block-based collaborative learning environment. Our findings revealed that sentiment polarity conflicts occurred in 31.8% of segments, with the highest frequency observed when all three modalities were present. Through theory-guided annotation, we categorized these conflicts into three types: (1) Affective Masking, (2) Suppressed Struggle, and (3) Full Disagreement. These categories are grounded in theories of Emotion Regulation, Control-Value, and Cognitive Load. Interestingly, these conflict types appeared with similar frequencies and correlated with independently annotated frustration levels. This indicates that disagreements between modalities capture emotionally significant states that are important for affect-aware, human-centered learning support.