Skip to content

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

Sep 2026 · arXiv.org · Vol abs/2609.36101 · 0 citations · 16 references
Computer Science

TL;DR

By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.

Abstract

Contrastive vision-language models learn shared embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal alignment, whereas removing gap-related structure can degrade image-text retrieval. In this paper, we provide a unified geometric explanation for these task-dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross-modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore performance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.

View source

Similar papers

Open access Sep 2026

Quantifying and reducing the modality gap in both contrastive and generative multimodal vision–language models

Dual-encoder vision-language models (VLMs) allow to embed texts and images in a shared representation space. However, it has been shown that these models are subject to a “modality gap phenomenon” meaning there exists a clear separation between the embeddings from one modality and another in the embedding space. While...

François Role, Sébastien Meyer, Victor Amblard · 0 citations
#computer vision Preprint Sep 2026

When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP

This work provides a systematic explanation of why modality gap reduction does not consistently improve CLIP zero-shot accuracy from the perspective of downstream decision structure and suggests that gap correction should be evaluated not only by average alignment, but also by its impact on downstream prediction struct...

Shota Sato, Hajime Kiyama, Tosho Hirasawa et al. · 0 citations
Preprint Sep 2026

UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport

UOT-Gap is introduced, a training-free variational diagnostic that models frozen image and text embeddings with unbalanced entropic optimal transport (UOT) and establishes UOT-Gap as a diagnostic for caption quality, modality alignment, and retrieval robustness.

Zong-Lin Yang, Hui-Lan Ma, Xu-Dan Zheng et al. · 0 citations
Review Aug 2026

Which Source Wins? Task-Dependent Reliance in Vision-Language Models

Modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, models, and evaluation settings, which shows that modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, and evaluation settings.

Rodela Ghosh, Aviral Gupta, Guang-Jing Wang · 0 citations

The Alignment Illusion in Multimodal Large Language Models

Internal visual-text alignment in MLLMs is best read as a geometric diagnostic of the visual stream inside the language model rather than a direct proxy for content-level cross-modal interaction, and is most informative when calibrated by controlled task evidence.

Hong-Han Wang, Yun-Tao Wang, Hui-Chao Ding · 0 citations
Aug 2026

Cross-modal alignment enhancement for lightweight large vision language models

A Low-Complexity Cross-Modal Alignment via Projection (LCAP) network is proposed, which introduces Projective Token Compression (PTC), which leverages Mish activation and adaptive average pooling to reduce feature redundancy while enhancing discriminative information, and Positional Spatial Enhancement (PSE), which exp...

Yu-Chen Sha, Lingli Wan, Ge Yang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.