Jul 2026· international journal of engineering trends and technology· 0 citations
TL;DR
Less obvious expressions like sarcasm and implicit sentiment are handled more effectively in this work, improving interpretation in multimodal sentiment analysis of social media data.
Abstract
The high rate of social media content development causes an increase in multimodal data, such that modeling relationships between visual and textual data is challenging. Nevertheless, most of the available methods cannot capture fine-grained text-to-visual or visual-to-text interaction, resulting in lower sentiment performance. A Contrastive Bidirectional Cross-Modal Attention (C-BCMA) model is presented to enhance the correspondence of textual and visual representations by acquiring a common latent space. An attention method inspired by CLIP is utilized to produce robust cross-modal latent features to enhance their joint representation. Textual features are derived using ALBERT, whereas EfficientNet-B2 is applied to obtain visual representations. Interactions between modalities are learned using a multi-head attention mechanism. Textual and visual information is handled jointly during learning. This helps reduce gaps between the two modalities. This enables the model to process various semantic cues at once. Contrastive learning is used in the model to align similar text-image pairs and to separate unrelated text-image pairs so that better multimodal representations are achieved. The model has a better performance than baseline approaches on both single and multiple annotation versions of MVSA datasets. It achieves better performance across various evaluation metrics. Less obvious expressions like sarcasm and implicit sentiment are handled more effectively in this work, improving interpretation in multimodal sentiment analysis of social media data.
MagXCL, a unified framework designed to improve multimodal integration through more effective interaction between verbal and non-verbal modalities, is proposed, demonstrating the effectiveness of combining AMag with CrossCL to produce more accurate and robust multimodal sentiment predictions.
As the amount of text and image data on social media continues to increase, multimodal sentiment analysis has emerged as a critical area of study. However, a "modality gap" that lowers classification accuracy is frequently caused by the domain disparities between textual and visual modalities. In this paper, a Translation Alignment method is used to improve a sentiment analysis system. Using three tools, first image captioning (BLIP), second facial expression identification (InsightFace), and last optical character recognition (EasyOCR), this method aligns the modalities by translating image information into textual descriptions. This transformation allows visual information to be represented in the same textual domain as the original post, reducing the semantic distance between modalities. In addition, the translated visual cues provide complementary information such as scene context, emotional expressions, and embedded text that may not be fully captured by the original caption alone. Using an Early Fusion strategy, the translated textual outputs from BLIP, InsightFace, and EasyOCR are concatenated with the original text prior to encoding, enabling the BERT-POS-LSTM architecture to process a unified multimodal textual representation. To rectify the MVSA dataset's imbalance, the SMOTE technique was applied to the latent feature space during training. The suggested model obtains an average accuracy of 73.7% and an F1 score of 72.6%, according to experimental results. These findings confirm that text-based domain alignment offers a more comprehensive and effective representation for multimodal sentiment analysis.
Uni-modal studies on sentiment analysis (SA) has made significant strides, emotions in the real world, which are typically multi-modal, encompassing audio, images, video and other contents more than text. The several modes contribute to mutual development. The accuracy of sentiment evaluation will be enhanced even more if it is possible to mine the connections between different modalities. This work presents a hybrid Deep Auto-Encoding with Convolutional Neural Networks (DAE+CNN), a multi-modal technique based on cross-attention based multi-modal fusion model for text and visuals. Text features are first extracted using the pre-training model; text context features are then extracted CNN; image characteristics are extracted using the network model; and specific emotion-related areas in images are extracted using DAE+CNN. The retrieved characteristics of the text and image are then fused using multi-modal cross-attention, and the output is classified to ascertain the emotional polarity. The model presented in this research outperforms the baseline model in the comparison study of the Fashion dataset and Deep Fashion datasets with accuracy of 86.5 % and 85.5% and F1-scores of 75.3% and 76.7%, respectively,. Furthermore, authors carried out ablation studies, which verified that multi-modal fusion sentiment evaluation outperforms single-modal sentiment analysis.
M. Yuvaraja, Dr. C. Kumuthini· Journal of Intelligent Decis...· 0 citations
DualScope is proposed, a novel model that combines a global-local fusion strategy with bidirectional image-text generation for semantically consistent data augmentation and introduces both label contrastive learning and data contrastive learning to align heterogeneous modalities and enhance model robustness.
Bing Zhang, Junteng Wang, Bin Sun et al.· Memetic Computing· 0 citations
Experimental evaluation on the PoliMemeDecode1 dataset shows that the attention-based fusion significantly outperforms unimodal baselines and standard concatenation methods, achieving a state-of-the-art Macro-F1 of approximately 0.94.
XSentiFusionNet is an end-to-end explainable multimodal framework for audio-visual sentiment analysis that incorporates cross-modal transformer-attention, reliability-aware adaptive fusion, and XAI and showed higher robustness to noise and generalization ability on different multimodal datasets.
M. Kidwai, C. Author, Dr. Faiyaz Ahmad· Journal of Intelligent Decis...· 0 citations