Retrv-MoE: Scaling Unified Multimodal Retrieval with Sparse Mixture-of-Experts

Abstract

Unified multimodal retrieval aims to build a single system capable of handling diverse modalities, tasks, and domains. While recent approaches leveraging multimodal large language models (MLLMs) have shown promise, they face a fundamental dilemma between capacity and interference: scaling dense models improves semantic understanding but incurs prohibitive inference costs, while training a single shared parameter space on heterogeneous data leads to severe gradient conflicts and negative transfer. In this work, we propose Retrv-MoE, a unified retrieval architecture built upon sparse Mixture-of-Experts (MoE). Unlike dense retrievers that activate all parameters for every input, Retrv-MoE employs learnable routers to dynamically select a small subset of experts for each token. We theoretically and empirically demonstrate that this conditional computation mechanism provides a structural remedy to optimization interference by decoupling the learning trajectories of conflicting tasks and domains into specialized expert subspaces. Extensive evaluations on the M-BEIR benchmark reveal that Retrv-MoE achieves a superior trade-off between efficiency and performance. It significantly outperforms efficiency-oriented baselines and matches the retrieval quality of 7B-parameter dense models while utilizing about 3 billion active parameters. Furthermore, our analysis confirms that the router exhibits emergent specialization, effectively mitigating negative transfer and enabling robust zero-shot generalization to unseen datasets and video retrieval tasks.