Unmasking Suppressed Emotion: A Multi-Agent Approach to Affective Dissonance

Abstract

Detecting true felt emotions when speakers suppress or mask their internal state poses a fundamental challenge for affective computing systems. We study a specific form of affective dissonance in dyadic speech: utterances where a speaker’s self-reported emotion diverges from all external observer ratings, indicating there might be emotional regulation. Using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) corpus, we identify 361 such utterances and propose a zero-shot multi-agent large language model (LLM) framework to infer the underlying felt state from audio, visual, and linguistic evidence. Our architecture deploys specialized agents that translate acoustic features (voice quality, pitch, rhythm, spectral energy) and visual signals (facial action unit proxies, head motion) into natural-language evidence reports, then integrates them via two fusion paradigms: a cascade design that aggregates per-modality predictions sequentially, and a direct design that presents raw feature descriptions from all modalities in a single LLM call, avoiding prediction anchoring. Experiments on the 361-utterance IEMOCAP subset demonstrate that our multi-agent framework substantially outperforms the random baseline (∼ 11.1%), with the best cascade condition achieving 20.9% accuracy and direct fusion reaching a Macro-F1 of 0.140. Notably, we find that linguistic content alone yields the strongest single-modality performance (Acc = 24.3%, Macro-F1 = 0.194), surpassing all fusion conditions, a result we interpret as a corpus artifact of IEMOCAP’s improvised scenario design, where scripted dialogue inflates the informativeness of transcribed text, pointing to the need for spontaneous, naturalistic corpora to advance affective dissonance research.