A multimodal solution that integrates audio and text information via cross-modal transformers, where text transcripts are automatically generated via an automatic speech recognition (ASR) tool and multiple text modalities are created via a cascaded architecture comprising cross-modal transformer blocks that integrate modalities one by one.
Abstract
Automatically recognizing the sentiment, positive or negative, from speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words. Recent solutions rely on audio foundation models to solve the task, but it remains unclear if such models can take all aspects into account. To this end, we propose a multimodal solution that integrates audio and text information via cross-modal transformers, where text transcripts are automatically generated via an automatic speech recognition (ASR) tool. Moreover, we create multiple text modalities by automatically translating the transcripts into multiple languages via machine translation tools. Audio and multilingual text features are combined via a cascaded architecture comprising cross-modal transformer blocks that integrate modalities one by one. We further distill knowledge from the multimodal model, called teacher, into a unimodal (audio only) model, called student. We conduct experiments on a large-scale dataset, demonstrating that the automatically generated textual information can bring significant performance boosts in multimodal sentiment polarity classification. Our ablation study confirms that both automatic transcripts and automatic translations are helpful. Moreover, we show that the audio-only model can be enhanced via distillation, boosting performance without any computational overhead during inference. To reproduce the reported results, we publicly release our code at https://github.com/andreidurdun/cross-modal-audio-sentiment.
MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples.
Shanshan Lin, Yuesheng Wu, Chao Chen et al.· 0 citations
HAFT routes audio–visual interaction through bottleneck tokens with adaptive depth-wise gating before integrating text, jointly reducing attention cost and counteracting modality bias; the Cascaded Audio Feature Enhancement (CAFE) framework strengthens prosodic representations via multi-scale time–frequency extraction; and grouped projections, decoupled positional attention, and layer-wise parameter sharing compress the remaining overhead.
Qing Dong, Ting Lu, Xiujin Shi et al.· International journal of sof...· 0 citations
A prototype-as-prompt framework that maps audio–visual representations into a fixed set of multimodal sentiment prototypes that are used as soft prompts to guide the LLM in performing MSA and introduces a sentiment-aware prototype learning that explicitly binds multimodal prototypes with sentiment semantics.
Xianbing Zhao, Lan Luo, Hengyang Lu et al.· 0 citations
MagXCL, a unified framework designed to improve multimodal integration through more effective interaction between verbal and non-verbal modalities, is proposed, demonstrating the effectiveness of combining AMag with CrossCL to produce more accurate and robust multimodal sentiment predictions.
As the amount of text and image data on social media continues to increase, multimodal sentiment analysis has emerged as a critical area of study. However, a "modality gap" that lowers classification accuracy is frequently caused by the domain disparities between textual and visual modalities. In this paper, a Translation Alignment method is used to improve a sentiment analysis system. Using three tools, first image captioning (BLIP), second facial expression identification (InsightFace), and last optical character recognition (EasyOCR), this method aligns the modalities by translating image information into textual descriptions. This transformation allows visual information to be represented in the same textual domain as the original post, reducing the semantic distance between modalities. In addition, the translated visual cues provide complementary information such as scene context, emotional expressions, and embedded text that may not be fully captured by the original caption alone. Using an Early Fusion strategy, the translated textual outputs from BLIP, InsightFace, and EasyOCR are concatenated with the original text prior to encoding, enabling the BERT-POS-LSTM architecture to process a unified multimodal textual representation. To rectify the MVSA dataset's imbalance, the SMOTE technique was applied to the latent feature space during training. The suggested model obtains an average accuracy of 73.7% and an F1 score of 72.6%, according to experimental results. These findings confirm that text-based domain alignment offers a more comprehensive and effective representation for multimodal sentiment analysis.
With the widespread growth of digital platforms, online interaction has become an essential part of everyday life. Users
frequently express their opinions, feedback, and emotions through reviews and comments on various platforms. Analyzing such
textual data plays a crucial role in understanding user sentiment and supporting effective decision-making. However, sentiment
analysis faces several challenges, including long-range dependencies within text and the presence of unknown words and
symbols. Traditional sentiment analysis approaches mainly rely on sequential models, which process text step by step and often
require higher computational time. In contrast, Transformer-based models offer improved efficiency through parallel
processing. To address these challenges, this paper presents a context-aware hybrid deep learning approach by integrating the
Robustly Optimized BERT Pretraining Approach (RoBERTa) with Bidirectional Long Short-Term Memory (BiLSTM) networks.
RoBERTa is employed to generate rich contextual word embeddings, while BiLSTM captures long-term semantic dependencies
by processing text in both forward and backward directions. The proposed model is trained and evaluated on the Twitter US
Airline Sentiment dataset comprising 14,299 samples across three sentiment classes. Experimental analysis demonstrates that
the hybrid approach achieves an accuracy of 85.14% and an F1-score of 0.8487, highlighting its effectiveness for sentiment
analysis tasks compared to baseline models
Dr. Veguru Gayatri, Dr. Rajani Rajalingam· International Journal for Re...· 0 citations