Multimodal Sentiment Analysis Through Deep Learning: Leveraging Early Fusion Using Translation Alignment
As the amount of text and image data on social media continues to increase, multimodal sentiment analysis has emerged as a critical area of study. However, a "modality gap" that lowers classification accuracy is frequently caused by the domain disparities between textual and visual modalities. In this paper, a Translation Alignment method is used to improve a sentiment analysis system. Using three tools, first image captioning (BLIP), second facial expression identification (InsightFace), and last optical character recognition (EasyOCR), this method aligns the modalities by translating image information into textual descriptions. This transformation allows visual information to be represented in the same textual domain as the original post, reducing the semantic distance between modalities. In addition, the translated visual cues provide complementary information such as scene context, emotional expressions, and embedded text that may not be fully captured by the original caption alone. Using an Early Fusion strategy, the translated textual outputs from BLIP, InsightFace, and EasyOCR are concatenated with the original text prior to encoding, enabling the BERT-POS-LSTM architecture to process a unified multimodal textual representation. To rectify the MVSA dataset's imbalance, the SMOTE technique was applied to the latent feature space during training. The suggested model obtains an average accuracy of 73.7% and an F1 score of 72.6%, according to experimental results. These findings confirm that text-based domain alignment offers a more comprehensive and effective representation for multimodal sentiment analysis.