Context-aware Multi-stream Adaptive CNN for Emotion Recognition

Abstract

Emotion recognition is a difficult task in the real world because of the interrelationship that exists between facial expression, scene setting, and social interaction.Conventional methods usually use single modality or simple fusion methods and do not provide rich multimodal relationships, and contextual dependencies.This study introduces a context-aware multistream adaptive network for emotion recognition, an integrated architecture that combines three independent streams: facial, scene, and interaction.The face stream uses a hybrid architecture based on ResNet-50 and an action unit graph neural network (AU-GNN).The scene stream expands CLIP with emotion-specific prompts to be learned for contextual semantic interpretation.The interaction stream applies a spatial-temporal graph attention network (ST-GAT) that uses social distance encoding to learn dynamic interpersonal relationships.These streams are combined using an Emotion Router, which modulates modality contributions adaptively depending on emotionspecific relevance and integrates uncertainty-compensated fusion to make effective predictions.The proposed model also predicts discrete emotion categories, continuous scores of valence-arousals, and uncertainty estimates.The experimental results demonstrate the state-of-the-art performance of the proposed algorithm.The proposed framework achieved an accuracy of 79.5% on the EMOTIC dataset, representing a 6.0 percentage point (pp) improvement over the baseline, and 86.9% on the CAER-S dataset, corresponding to a 6.7 pp improvement over the baseline.On the AFEW-VA dataset, it achieved a Concordance Correlation Coefficient (CCC) of 0.574.Experimental analyses confirmed the usefulness and decipherability of the proposed framework.