Rebalancing Multimodal Learning via Adaptive Bidirectional Alignment
Abstract
Multimodal learning often suffers from modality imbalance, leading to an optimization dilemma. A key reason is that different modalities exhibit asynchronous learning dynamics during joint training. A representative line of existing methods seeks to rebalance multimodal learning through cross-modal alignment. However, such alignment is usually imposed in a static, symmetric, and fully coupled manner, making it prone to unreliable cross-modal interactions. To address this issue, we argue that cross-modal alignment under modality imbalance should be space-aware and status-aware, reducing feature-space discrepancy while selectively aligning transferable components according to modality learning status. Based on this principle, we propose an Adaptive Bidirectional Alignment (ABA) method. ABA first constructs class-conditional statistical projections to establish more robust semantic correspondences between heterogeneous modality spaces. It then employs feature-level partial optimal transport to selectively align reliable and transferable feature components. Furthermore, we introduce a dynamic bidirectional regulation mechanism to adaptively adjust the direction and strength of cross-modal knowledge transfer according to modality learning status, enabling more effective cross-modal collaboration. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SOTA) multimodal learning baselines. The source code is available at https://github.com/yilnr/ABA our repository