Skip to content
Open access

Quantifying and reducing the modality gap in both contrastive and generative multimodal vision–language models

Sep 2026 · International Journal of Multimedia Information Retrieval · Vol 15 · 0 citations · 30 references
Computer Science

Abstract

Dual-encoder vision-language models (VLMs) allow to embed texts and images in a shared representation space. However, it has been shown that these models are subject to a “modality gap phenomenon” meaning there exists a clear separation between the embeddings from one modality and another in the embedding space. While this misalignment is detrimental for downstream tasks such as multimodal retrieval, multimodal clustering or zero-shot classification, no generic and practical methods have so far been proposed to assess it precisely and even reduce it. We therefore propose novel measures and an effective spectral technique to achieve this goal, noting that, in its basic form, the spectral method offers no out-of-sample extension – although a Nyström-style approximation could provide one – and mainly targets offline retrieval settings. On the other hand, a new breed of model has emerged, that can handle combined-modal inputs to a multimodal large language model, and extract a unified, fused vector representation. These models open up new research questions about possible biases of these fused representations towards a specific modality, and should such biases be found to exist, if they can be reduced drawing on the same techniques utilized for dual-encoder models.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.