Quantifying and reducing the modality gap in both contrastive and generative multimodal vision–language models
Dual-encoder vision-language models (VLMs) allow to embed texts and images in a shared representation space. However, it has been shown that these models are subject to a “modality gap phenomenon” meaning there exists a clear separation between the embeddings from one modality and another in the embedding space. While...