Emotion-BIND for multimodal emotion recognition and reasoning.
TL;DR
Empir-BIND is presented, a multimodal framework that integrates ImageBind-based representations with a differentiated positional encoding strategy, termed m-ROPE, and applies 2D RoPE to image representations and 3D RoPE to video representations, supporting multimodal spatial and temporal modeling.
Abstract
Constructing Multimodal Emotion Recognition in Conversation (MERC) models is important for understanding affective states from text, audio, images, and video. Existing approaches often rely on linear layers for cross-modal feature alignment and freeze encoder parameters during training, which can introduce feature degradation and modality misalignment. Conventional one-dimensional positional encoding is also limited when modeling visual and temporal information. To address these issues, this study presents Emotion-BIND, a multimodal framework that integrates ImageBind-based representations with a differentiated positional encoding strategy, termed m-ROPE. The framework maps features from multiple modalities into a shared embedding space and applies 2D RoPE to image representations and 3D RoPE to video representations, supporting multimodal spatial and temporal modeling. Experimental results show that Emotion-BIND achieves an Unweighted Average Recall (UAR) of 49.44%, a Weighted Average Recall (WAR) of 71.00%, and an F1 score of 90.05% on the DFEW dataset, with the highest WAR and F1 score among the compared methods, although its UAR remains lower than the best comparison model. On the MELD dataset, Emotion-BIND achieves a UAR of 55.67%, a WAR of 67.88%, and an F1 score of 91.02%. The source code is archived at https://doi.org/10.5281/zenodo.21186484 and is also available on GitHub at https://github.com/qunlitang/Emotion-BIND.