Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval
Abstract
With the proliferation of media data across diverse sources, multimodal information retrieval has emerged as a trending research. However, it faces inherent challenges arising from heterogeneous data sources and complex cross-modal alignment. Although previous studies have revealed modal gaps in feature spaces, a systematic approach to address these challenges remains unexplored. In this work, we introduce Unite, a universal multimodal embedding framework that jointly supports text, image, and video modalities. We systematically investigate two critical yet underexplored aspects: data curation and modality-aware training configurations. Specifically, we provide a comprehensive analysis of how modality-specific data properties (e.g., cross-modal distributional discrepancies, varying levels of semantic granularity) systematically influence downstream task performance across diverse scenarios. Furthermore, we propose Modal-Aware Masked Contrastive Learning (MAMCL), which selectively masks non-target-modal instances based on modality-aware signals, enabling the model to learn more discriminative representations and effectively mitigating cross-modal competition. Extensive experiments on diverse multimodal retrieval benchmarks show that Unite achieves state-of-the-art performance across coarse-grained, fine-grained, and instruction-based retrieval tasks. These results demonstrate that strategic modality-specific data curation and tailored training protocols are pivotal for robust universal multimodal representation learning.