Skip to content
Book Open access

Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

May 2025 · Proceedings of the 34th ACM International Conference on Multimedia · 30 citations · ⚡ 5 influential · 124 references
Computer Science

Abstract

With the proliferation of media data across diverse sources, multimodal information retrieval has emerged as a trending research. However, it faces inherent challenges arising from heterogeneous data sources and complex cross-modal alignment. Although previous studies have revealed modal gaps in feature spaces, a systematic approach to address these challenges remains unexplored. In this work, we introduce Unite, a universal multimodal embedding framework that jointly supports text, image, and video modalities. We systematically investigate two critical yet underexplored aspects: data curation and modality-aware training configurations. Specifically, we provide a comprehensive analysis of how modality-specific data properties (e.g., cross-modal distributional discrepancies, varying levels of semantic granularity) systematically influence downstream task performance across diverse scenarios. Furthermore, we propose Modal-Aware Masked Contrastive Learning (MAMCL), which selectively masks non-target-modal instances based on modality-aware signals, enabling the model to learn more discriminative representations and effectively mitigating cross-modal competition. Extensive experiments on diverse multimodal retrieval benchmarks show that Unite achieves state-of-the-art performance across coarse-grained, fine-grained, and instruction-based retrieval tasks. These results demonstrate that strategic modality-specific data curation and tailored training protocols are pivotal for robust universal multimodal representation learning.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.