Experimental evaluation on the PoliMemeDecode1 dataset shows that the attention-based fusion significantly outperforms unimodal baselines and standard concatenation methods, achieving a state-of-the-art Macro-F1 of approximately 0.94.
Abstract
Automated analysis of multimodal content on social networks has become a critical task for understanding public sentiment and information diffusion in the digital age. However, classifying internet memes remains computationally challenging due to the intricate interplay between visual cues and embedded, often stylized, text, particularly in low-resource languages like Bengali Language. This paper addresses the detection of political intent in Bengali memes by introducing Multimodal Cross-Attention Fusion framework. We first leverage a Vision-Language Model to extract high-fidelity OCR text from noisy meme images. Subsequently, we encode visual and textual features and synthesize them through a cross-modal multi-head attention mechanism that aligns semantic tokens with visual regions. We also investigate the integration of a domain-specific political lexicon as a knowledge prior. Experimental evaluation on the PoliMemeDecode1 dataset shows that our attention-based fusion significantly outperforms unimodal baselines and standard concatenation methods, achieving a state-of-the-art Macro-F1 of approximately 0.94. Interpretability analyzes further confirm that the model effectively learns to ground textual semantics in visual evidence.
Less obvious expressions like sarcasm and implicit sentiment are handled more effectively in this work, improving interpretation in multimodal sentiment analysis of social media data.
Prashant Adakane, Amit Gaikwad· international journal of eng...· 0 citations
XSentiFusionNet is an end-to-end explainable multimodal framework for audio-visual sentiment analysis that incorporates cross-modal transformer-attention, reliability-aware adaptive fusion, and XAI and showed higher robustness to noise and generalization ability on different multimodal datasets.
M. Kidwai, C. Author, Dr. Faiyaz Ahmad· Journal of Intelligent Decis...· 0 citations
MagXCL, a unified framework designed to improve multimodal integration through more effective interaction between verbal and non-verbal modalities, is proposed, demonstrating the effectiveness of combining AMag with CrossCL to produce more accurate and robust multimodal sentiment predictions.
MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples.
Shanshan Lin, Yuesheng Wu, Chao Chen et al.· 0 citations
The increasing availability of social networking sites has ensured that memes have become a part of our daily lives to express our opinion through communication. Images or plain text are in contrast to multimodal content consisting of textual captions on images to explain emotions, opinions, and sarcasm in implicit and in some cases ambiguous ways. Identification and classification of these emotions are not a straightforward task but demand robust multimodal learning models that can glue visual as well as textual information together. This paper presents an end-to-end multimodal emotion analysis model based on the MEMotion dataset, integrating state-of-the-art deep learning structures, strict preprocessing, and extensive evaluation approaches to confront the difficulty of meme-based emotion detection. The proposed system is a formal multi-stage pipeline. Stage one delivers dataset refinement through cleaning, path correction, binarization of labels, and stratified splitting to balance classes in training and validation sets. Stage two is a hybrid approach in which ResNet-based convolutional networks learn discriminative visual features, while BERT encoders learn contextualized textual semantics. The features are then fused through dense projection layers and dropout-normalized to prevent overfitting. Stage three involves adaptive training mechanisms with the application of weighted cross-entropy, focal loss, and their hybrid forms for handling extreme class imbalance across multiple labels: humour, sarcasm, offensive, motivational, and general sentiment. Stage four enhances experimental robustness with gradient accumulation, learningrate warmup, early stopping, and misclassification grids. A broad set of per-task confusion matrix, aggregated confusion grid, ROC and precision-recall curve, loss–accuracy plot, and per-class accuracy plot outputs were generated to enable more in-depth model performance interpretability.
I. S, F. H. Fernandez· 2026 7th International Confe...· 0 citations
An Aspect-guided dual-branch fusion network (ADFN) to enhance sentiment prediction by incorporating external knowledge and integrating coarse and fine information is proposed, which incorporates syntactic dependency information to complement and enrich the textual semantic representations.
Bin Song, Wenjing Liu, Zhipeng Liang et al.· Signal, Image and Video Proc...· 0 citations