Accelerating Feature Aggregation for Imbalanced Multimodal Learning
Abstract
Imbalanced multimodal learning remains a critical challenge, as models often over-rely on dominant modality while under optimizing non-dominant one, ultimately limiting overall performance. Essentially, we observe that such over-reliance is closely associated with discrepancies in feature aggregation capability across modalities. More concretely, non-dominant modality lags in forming compact and stable representations, whereas dominant modality exhibits superior aggregation capability. To address this issue, we propose a simple yet effective approach called Dynamic Sample Retrieval for Efficient Feature Aggregation (DSR-EFA). Specifically, we first introduce a retrieval-driven near-neighbor sample augmentation strategy to build a candidate pool for each modality. Based on these pools, we propose a modality-aware dynamic ensemble scheme that allocates retrieval budget according to modality importance and constructs additional informative samples for classification. Consequently, DSR-EFA effectively reduces feature aggregation disparities and improves multimodal learning performance. Extensive experiments on widely used datasets verify the effectiveness of DSR-EFA.