GTPF: Modality-specific graph transformer with prompt-aware fusion for multimodal sentiment analysis.

Abstract

Multimodal Sentiment Analysis (MSA) integrates information from multiple modalities to infer sentiment. It faces two challenges: accurately modeling unimodal modalities and effectively fusing them. Existing graph neural network (GNN) based methods are limited by the over-smoothing problem caused by deep architectures, and thus typically adopt shallow structures for modality modeling. While these shallow structures can capture local information within a modality well, they struggle to capture global context. Meanwhile, current text-centric fusion methods do not explicitly align textual and non-textual modalities before fusion, which degrades fusion performance. To address these issues, we propose a modality-specific Graph Transformer with Prompt-aware Fusion (GTPF) framework. GTPF employs a Modality-Specific Graph Transformer (MSGT) architecture to explore both local and global information within modalities, enabling more accurate unimodal modeling. It also uses a Prompt-Aware Multimodal Fusion (PAMF) module that adopts prompt shifting to align textual and non-textual modalities before fusion, thereby enhancing text-centric fusion. We conduct extensive experiments on three MSA benchmarks. GTPF outperforms state-of-the-art methods across all metrics. On the CMU-MOSI dataset, GTPF achieves a relative improvement of up to 13.76% in mean absolute error (MAE) over the second-best model. On the CH-SIMS dataset, it achieves relative improvements of 6.01% in Pearson correlation (Corr) and 5.03% in five-class classification accuracy over the second-best model. These results validate the effectiveness of our Graph-Transformer co-design and prompt-aware fusion strategy.