DART: dynamic adapter refinement at test-time for multimodal document retrieval

Abstract

While Multimodal Large Language Models (MLLMs) exhibit strong capabilities in document retrieval, their zero-shot performance is frequently bottlenecked by a static inference paradigm. Once deployed, these frozen models struggle to dynamically adjust their representation spaces to accommodate highly diverse query semantics and intricate visual layouts. To overcome this inherent rigidity, we introduce Dynamic Adapter Refinement at Test-time (DART), a novel inference-stage adaptation strategy. Rather than fine-tuning the massive backbone or applying destructive global feature projections, DART injects a lightweight, residual Adapter network to re-rank the initial top-K candidates. Our method extracts soft pseudo-labels directly from the model's initial retrieval confidence to drive a self-supervised contrastive learning objective. Crucially, to prevent catastrophic representation drift and preserve the rich vision-language alignment acquired during pre-training, DART constrains the Adapter's parameter updates using a targeted 𝐿𝐿2regularization penalty. Empirical evaluations across a diverse suite of multimodal document retrieval benchmarks reveal that DART achieves consistent and significant gains in ranking precision. Furthermore, this dynamic refinement process introduces minimal computational latency, offering a highly efficient, plug-and-play solution for adaptive document retrieval.