DiMoE: Disentangled Representation Learning with Mixture of Experts Fusion for Sentiment Intensity Prediction and Emotion Classification
TL;DR
DiMoE is proposed, a representation-first framework that integrates feature disentanglement with sparse Mixture of Experts (MoE) fusion, indicating that explicit modality disentanglement combined with Mixture of Experts fusion yields effective multimodal affective representations.
Abstract
Multimodal affective analysis benefits from combining textual, acoustic, and visual cues, yet many methods implicitly mix modality-invariant information with modality-specific factors, which can reduce robustness when modalities vary in reliability. We propose DiMoE, a representation-first framework that integrates feature disentanglement with sparse Mixture of Experts (MoE) fusion. DiMoE first decomposes each modality into shared (modality-invariant) and private (modality-specific) representations. It then fuses these factors using top-k sparse MoE routing, enabling input-adaptive expert selection. We study two routing strategies with a shared expert pool, a joint router that operates on combined shared and private factors, and separate routers that process shared and private factors independently. We evaluate DiMoE on five widely used benchmarks spanning sentiment intensity prediction and emotion classification, covering both trimodal and bimodal settings. Across benchmarks, DiMoE consistently achieves strong performance against recent competitive baselines, indicating that explicit modality disentanglement combined with Mixture of Experts fusion yields effective multimodal affective representations.