Multimodal Sentiment Analysis via Modal Experts and Missing-Modal Prompt Generation

Abstract

Multimodal sentiment analysis aims to extract affective signals from multiple modalities and perform feature analysis, modality fusion and sentiment prediction. However, in practical application scenarios, input modality information is highly prone to be missing due to various objective factors, which further causes the model to produce biased or even completely erroneous sentiment judgment results. To address this critical issue, a model named Modal Experts and Missing Prompt Generation (MEMPG) is proposed. The model employs a brand-new dual-residual modality expert architecture to integrate the knowledge of hybrid modality experts. This architecture can not only retain the original information but also preserve the contextual information learned by the attention mechanism, thus exhibiting better robustness. The model adopts a twostage training strategy: In the first stage, each modality expert branch is independently pre-trained to extract low-level basic features of a single modality and strengthen the feature extraction capability of modality experts. In the second stage, all input modality features are fed into all modality expert branches for processing, and adaptive weights are assigned to modality features via an adaptive gating mechanism to obtain fused modality features with stronger representation ability. Subsequently, the enhanced features are input into the missing modality prompt generation module, which incorporates prompt learning to guide the reconstruction of missing modalities. Extensive comparative experiments are conducted on two standard multimodal sentiment datasets, namely MOSI and MOSEI, under various common modality missing scenarios. Experimental results demonstrate that, compared with current mainstream baseline models, the proposed MEMPG model achieves superior sentiment prediction performance.