Multimodal information fusion analysis based on deep learning

Abstract

In the social media ecosystem, user-generated content has gradually evolved into a multimodal form coexisting with text and speech. A single information dimension can hardly fully characterize users' opinions and emotional tendencies. To address the problems of insufficient feature representation in unimodal sentiment recognition and significant information interference in traditional fusion methods, a mid-term feature fusion multimodal analysis model combining Text Convolutional Neural Network (TextCNN) and Bidirectional Long Short-Term Memory (BiLSTM) is constructed. The model relies on a dual-branch structure to mine text semantic features and speech timing features respectively, completes cross-modal information complementary fusion through feature concatenation, and connects a fully connected layer and Softmax classifier to realize three-class sentiment discrimination. Simulation experiments based on the CMU-MOSEI dataset show that the model achieves an accuracy of 0.74 and a Macro-F1 value of 0.73, which is 2.8 percentage points higher in accuracy and 3.1 percentage points higher in Macro-F1 value than the late fusion model. Ablation experiments confirm that the text branch, speech branch and feature fusion unit all provide positive support for model performance. The mid-term fusion method of multimodal features can effectively make up for the defects of unimodal information, providing a feasible technical solution for multimodal sentiment recognition and information mining in social scenarios.