TL;DR
A novel method called Prompt Alignment and Multi-Granularity Feature Fusion (PAMFF), which employs prompt templates to construct prompts for aspect terms and then generates soft entity pseudo-labels derived from both the prompt features and the image entity features to achieve deeper cross-modal information fusion.
MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples.
An Aspect-guided dual-branch fusion network (ADFN) to enhance sentiment prediction by incorporating external knowledge and integrating coarse and fine information is proposed, which incorporates syntactic dependency information to complement and enrich the textual semantic representations.
Multimodal Sentiment Analysis (MSA) integrates information from multiple modalities to infer sentiment. It faces two challenges: accurately modeling unimodal modalities and effectively fusing them. Existing graph neural network (GNN) based methods are limited by the over-smoothing problem caused by deep architectures, and thus typically adopt shallow structures for modality modeling. While these shallow structures can capture local information within a modality well, they struggle to capture global context. Meanwhile, current text-centric fusion methods do not explicitly align textual and non-textual modalities before fusion, which degrades fusion performance. To address these issues, we propose a modality-specific Graph Transformer with Prompt-aware Fusion (GTPF) framework. GTPF employs a Modality-Specific Graph Transformer (MSGT) architecture to explore both local and global information within modalities, enabling more accurate unimodal modeling. It also uses a Prompt-Aware Multimodal Fusion (PAMF) module that adopts prompt shifting to align textual and non-textual modalities before fusion, thereby enhancing text-centric fusion. We conduct extensive experiments on three MSA benchmarks. GTPF outperforms state-of-the-art methods across all metrics. On the CMU-MOSI dataset, GTPF achieves a relative improvement of up to 13.76% in mean absolute error (MAE) over the second-best model. On the CH-SIMS dataset, it achieves relative improvements of 6.01% in Pearson correlation (Corr) and 5.03% in five-class classification accuracy over the second-best model. These results validate the effectiveness of our Graph-Transformer co-design and prompt-aware fusion strategy.
DualScope is proposed, a novel model that combines a global-local fusion strategy with bidirectional image-text generation for semantically consistent data augmentation and introduces both label contrastive learning and data contrastive learning to align heterogeneous modalities and enhance model robustness.
In multimodal sentiment analysis, textual, acoustic, and visual modalities often contain redundant and noisy information. Such information increases model complexity and weakens core sentiment representations, degrading accuracy and robustness. To address this issue, we propose CLIBN, a multimodal sentiment recognition network based on contrastive learning and information bottleneck. First, we design a sentimentintensity-aware contrastive learning strategy. It constructs positive and negative pairs according to sentiment intensity distances and assigns adaptive weights to different pairs, enabling the model to capture fine-grained sentiment differences. Second, we introduce a hierarchical information bottleneck module. It treats text as the primary modality and progressively integrates complementary cues from acoustic and visual modalities, while preserving task-relevant semantics and suppressing redundant information. Experimental results on CMU-MOSI and CMU-MOSEI show that CLIBN achieves superior performance. Specifically, Acc-2 reaches 87.8% and 86.7%, and F1-Score reaches 87.8% and 86.6% on the two datasets, respectively. These results demonstrate the effectiveness of CLIBN for multimodal sentiment representation learning.
Multimodal sentiment analysis has attracted increasing attention due to the prevalence of text-image content on social media. A central challenge is to design fusion mechanisms that are both expressive and parameter-efficient, especially for small-scale datasets where heavy cross-modal attention can easily overfit. In this paper, we present Consistency-Aware Gated Fusion (CAGF), a lightweight and fusion module tailored to Mamba-based architectures. Our key idea is to exploit Mamba's bidirectional scanning mechanism: forward and backward hidden states from text and image encoders are concatenated to form enhanced representations, and a cosine-based semantic consistency score is computed between modalities. This score is then passed through a fixed sigmoid gate to adaptively weight text and image features, without introducing any additional learnable parameters. CAGF is plug-and-play compatible with dual-stream Mamba encoders and incurs negligible computational overhead compared with attention-based fusion. Experiments on the MVSA-Single dataset show that CAGF achieves state-of-the-art performance (Acc=82.54%, F1=84.82%), outperforming strong multimodal baselines such as CLIP, MISA, DLF, AoM, and SFTTR, while remaining more efficient and interpretable. Extensive ablations and sensitivity analyses further validate that bidirectional scanning, enhanced representations, and consistency-aware gating are all critical to the observed gains.