Jul 2026· International Conference on Information and Communicatiaon Technology· pp. 1-6· 0 citations· 21 references
Abstract
As the amount of text and image data on social media continues to increase, multimodal sentiment analysis has emerged as a critical area of study. However, a "modality gap" that lowers classification accuracy is frequently caused by the domain disparities between textual and visual modalities. In this paper, a Translation Alignment method is used to improve a sentiment analysis system. Using three tools, first image captioning (BLIP), second facial expression identification (InsightFace), and last optical character recognition (EasyOCR), this method aligns the modalities by translating image information into textual descriptions. This transformation allows visual information to be represented in the same textual domain as the original post, reducing the semantic distance between modalities. In addition, the translated visual cues provide complementary information such as scene context, emotional expressions, and embedded text that may not be fully captured by the original caption alone. Using an Early Fusion strategy, the translated textual outputs from BLIP, InsightFace, and EasyOCR are concatenated with the original text prior to encoding, enabling the BERT-POS-LSTM architecture to process a unified multimodal textual representation. To rectify the MVSA dataset's imbalance, the SMOTE technique was applied to the latent feature space during training. The suggested model obtains an average accuracy of 73.7% and an F1 score of 72.6%, according to experimental results. These findings confirm that text-based domain alignment offers a more comprehensive and effective representation for multimodal sentiment analysis.
Less obvious expressions like sarcasm and implicit sentiment are handled more effectively in this work, improving interpretation in multimodal sentiment analysis of social media data.
Prashant Adakane, Amit Gaikwad· international journal of eng...· 0 citations
Uni-modal studies on sentiment analysis (SA) has made significant strides, emotions in the real world, which are typically multi-modal, encompassing audio, images, video and other contents more than text. The several modes contribute to mutual development. The accuracy of sentiment evaluation will be enhanced even more if it is possible to mine the connections between different modalities. This work presents a hybrid Deep Auto-Encoding with Convolutional Neural Networks (DAE+CNN), a multi-modal technique based on cross-attention based multi-modal fusion model for text and visuals. Text features are first extracted using the pre-training model; text context features are then extracted CNN; image characteristics are extracted using the network model; and specific emotion-related areas in images are extracted using DAE+CNN. The retrieved characteristics of the text and image are then fused using multi-modal cross-attention, and the output is classified to ascertain the emotional polarity. The model presented in this research outperforms the baseline model in the comparison study of the Fashion dataset and Deep Fashion datasets with accuracy of 86.5 % and 85.5% and F1-scores of 75.3% and 76.7%, respectively,. Furthermore, authors carried out ablation studies, which verified that multi-modal fusion sentiment evaluation outperforms single-modal sentiment analysis.
M. Yuvaraja, Dr. C. Kumuthini· Journal of Intelligent Decis...· 0 citations
Sentiment analysis is essential for several real-world applications, such as opinion mining and predicting a person's intent and personality. Most existing work aims to address challenges of sentiment analysis using normal text and images uploaded on social media. This work aims to use scene text images for sentiment analysis to assist in understanding the intentions of captured scenes. We present TSRB (Transformer-based Semantic Refinement Block), which comprises a multimodal approach and semantic gating. The proposed method constructs hierarchically fused image and text representations and then routes them through a TSRB and a learned three-way Semantic Gating module. The image branch encodes both the full meme image and text image extracted from the input image through a convolutional network with spatial attention; the text branch encodes OCR text, raw tweet text, and image captions via three independent Distil-BERT+CNN encoders and hierarchically fuses them. The resulting visual and textual embeddings are jointly refined by three stacked Transformer encoder layers within the proposed TSRB and then selectively blended by a softmax-weighted Semantic Gate that dynamically arbitrates among the post-attention, visual, and textual streams. Experiments are conducted on two standard datasets (MVSA-Single and Memotion) and compared with state-of-the-art models to demonstrate the effectiveness of the proposed method.
Soutik Mukherjee, Shivakumara Palaiahnakote, Umapada Pal et al.· International journal of pat...· 0 citations
Image–text multimodal sentiment analysis aims to integrate textual and visual information to comprehensively understand sentiment expressions in complex scenarios. However, existing methods focus on cross-modal feature interaction and fusion, and still have difficulty capturing effective sentiment cues in scenarios involving insufficient textual semantics, implicit visual affective cues, and inconsistent sentiment expressions between text and image. To address these issues, this paper proposes an image–text multimodal sentiment analysis method with large model-generated descriptive semantics and difference-aware gated fusion. Specifically, a large model generates semantic descriptions for image–text pairs, from which an enhanced semantic view is constructed to supplement implicit or insufficiently expressed sentiment cues in the original modalities. An original-enhanced dual-branch structure models the original image–text evidence and enhanced semantic evidence separately. To improve semantic consistency between the two branches, a cross-branch semantic alignment mechanism is introduced to reduce semantic shifts caused by enhanced information. In the fusion stage, difference-aware gated fusion and residual compensation are employed to adaptively balance branch contributions while preserving discriminative branch differences. Experimental results on the MVSA-Single and MVSA-Multiple datasets show that the proposed method improves performance in image–text multimodal sentiment classification, thereby validating the effectiveness of combining semantic enhancement with difference-aware modeling.
Hengyuan Zhang, Aizihaierjiang Yusufu, Jiang Liu et al.· Electronics· 0 citations
MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples.
Shanshan Lin, Yuesheng Wu, Chao Chen et al.· 0 citations
DualScope is proposed, a novel model that combines a global-local fusion strategy with bidirectional image-text generation for semantically consistent data augmentation and introduces both label contrastive learning and data contrastive learning to align heterogeneous modalities and enhance model robustness.
Bing Zhang, Junteng Wang, Bin Sun et al.· Memetic Computing· 0 citations