This work proposes LLM-Augmented Prompt Learning for Multimodal Sentiment Analysis with Reward Adaptation (LAPM-RA), a unified framework integrating LLM-based sentiment-aware augmentation, reward-guided prompt selection, and context-aware multimodal fusion.
Abstract
Few-shot multimodal sentiment analysis (MSA), constrained by limited annotated data, often suffers from large cross-modal semantic alignment gaps and difficulty in capturing fine-grained sentiment cues, making it challenging to fully exploit the complementary information between text and images. Recent advances in large language models (LLMs) and prompt learning have shown strong potential for improving label efficiency in low-resource natural language processing tasks; however, their direct application to MSA is hindered by static prompt designs and shallow cross-modal integration. To overcome these limitations, we propose LLM-Augmented Prompt Learning for Multimodal Sentiment Analysis with Reward Adaptation (LAPM-RA), a unified framework integrating LLM-based sentiment-aware augmentation, reward-guided prompt selection, and context-aware multimodal fusion. Specifically, LLMs generates sentiment-consistent and counterfactual text variants to enhance lexical and structural diversity while preserving label fidelity; a supervised policy network adaptively selects optimal prompt templates based on reward signals; and a lightweight gating mechanism integrates textual and visual embeddings contextually. Extensive experiments on multiple benchmarks validate the effectiveness and robustness of LAPM-RA over competitive baselines.
MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples.
Shanshan Lin, Yuesheng Wu, Chao Chen et al.· 0 citations
A prototype-as-prompt framework that maps audio–visual representations into a fixed set of multimodal sentiment prototypes that are used as soft prompts to guide the LLM in performing MSA and introduces a sentiment-aware prototype learning that explicitly binds multimodal prototypes with sentiment semantics.
Xianbing Zhao, Lan Luo, Hengyang Lu et al.· 0 citations
: Multimodal sentiment analysis (MSA) has made significant progress in integrating heterogeneous information from text, speech, and vision. However, real-world multimodal data often suffer from modality noise, semantic inconsistency, and incomplete modality information, which can weaken cross-modal fusion and reduce the reliability of sentiment prediction. To address these challenges, this paper proposes RUAL, a robust uncertainty-aware learning framework for multimodal sentiment analysis. Specifically, RUAL first employs a Gathered Multi-Head Attention Pooling (GMHA) module to aggregate intra-modal features and estimate modality uncertainty based on attention entropy. Then, an Uncertainty-Aware Cross-Modal Coupled Layer (UACCL) is introduced to dynamically regulate cross-modal residual fusion according to sample confidence, thereby reducing the negative influence of unreliable modalities on fused representations. In addition, uncertainty-weighted learning and uncertainty-guided self-distillation (UWL and U-SD) are jointly integrated through an optimization strategy to further improve training stability and generalization in complex scenarios. Experimental results on CMU-MOSI, CMU-MOSEI, and MVSA-Single demonstrate that RUAL achieves strong overall performance and maintains stable prediction results under missing-modality and Gaussian-noise conditions, validating the effectiveness and robustness of the proposed framework for multimodal sentiment analysis.
Multimodal aspect-based sentiment analysis (MABSA) predicts the sentiment polarity associated with a specified aspect by jointly exploiting textual and visual information. Existing models may be sensitive to limited prompting examples, cross-modal noise, and unreliable generated context. This paper proposes QMPN, Quality-Aware Memory Prompting Network, that stores sample-specific prompts derived from a small support set, retrieves relevant prompting evidence for each query, and uses the retrieved prompts to guide aspect-aware context generation. A task-oriented quality gate, learned indirectly through the sentiment classification objective, controls the contribution of the generated context to the final prediction. Under the fixed protocol used in this study, QMPN employs 50 labeled support instances for prompt-memory construction and achieves 78.6% accuracy and 74.8% macro-F1 on Twitter-2015, and 72.0% accuracy and 70.5% macro-F1 on Twitter-2017. Relative to the variant without context generation, the complete model improves accuracy/macro-F1 by 3.31/2.83 percentage points on Twitter-2015 and 3.84/3.22 percentage points on Twitter-2017. Ablation and parameter analyses further show the contributions of memory retrieval, adaptive prompt selection, context generation, and quality-aware fusion. Because the evaluation is limited to two historical Twitter benchmarks and a single fixed seed, the reported results should be interpreted within this experimental scope rather than as evidence of universal cross-domain generalization.
Lei Pan, Tong Geng, Yuheng Liu· Information· 0 citations
Incomplete multimodal sentiment analysis has garnered significant attention in recent years. Existing approaches typically assume that data is missing at random or are designed specifically for certain missing patterns, ignoring the modality combination inconsistency between training and testing phases. However, in real-world scenarios, the testing phase often encounters modal combinations that were not present during the training phase, which leads to insufficient generalization capabilities and unstable performance. In this paper, we introduce the problem of Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC), aiming to enhance model generalization for unseen modality combinations. To address this challenge, we propose the model named $\textbf{C}$ontrastive $\textbf{M}$ixed $\textbf{P}$rompt $\textbf{L}$earning ($\textsf{CMPL}$) for IMSAUMC. It introduces a label-guided contrastive feature learning mechanism to learn robust and discriminative cross-modal representations. Additionally, we design modality-combination prompts with a soft router to facilitate better learning of various modality combinations. Furthermore, we introduce three prompt contrastive learning strategies, which enable effective learning of prompts corresponding to unseen modality combinations, thereby significantly strengthening the model's generalization capabilities in diverse testing scenarios. Extensive experiments on three widely used datasets demonstrate that $\textsf{CMPL}$ achieves more than a 5% improvement in accuracy compared to state-of-the-art approaches.
Kaixin Xu, NaiJin Liu, Yu-Ri Kang et al.· 0 citations