Skip to content

How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

Sep 2026 · arXiv.org · Vol abs/2609.36557 · 0 citations · 48 references
Computer Science

TL;DR

A mechanistic analysis of the VLM model's internal attention patterns shows that a simple describe-then-decide prompting strategy increases vision attention by 30-40% during generation, and task-specific fine-tuning improves dermatology classification but reduces cross-domain medical question-answering performance in the authors' evaluation.

Abstract

Medical Vision-Language Models (VLMs) show significant promise for clinical image understanding, offering accurate diagnosis with interpretable reasoning. However, a critical performance gap exists between their strong vision encoders and the full multimodal model: in dermatology, the MedSigLIP encoder outperforms MedGemma by an average of 10.26 percentage points even when both use zero target-task labels; few-shot linear probing provides further evidence of strong visual representations. This gap motivates an investigation of how visual information is used in end-to-end diagnosis and why plausible-sounding predictions can lack grounding in image evidence. Using dermatology as our primary testbed, we systematically investigate three hypotheses for this phenomenon. We further provide a mechanistic analysis of the model's internal attention patterns, showing that a simple describe-then-decide prompting strategy increases vision attention by 30-40% during generation. Task-specific fine-tuning improves dermatology classification but reduces cross-domain medical question-answering performance in our evaluation. To address these challenges, we combine label-free prompting with low-label encoder-assisted reranking while keeping the VLM frozen. We validate the interventions across five VLM backbones in dermatology and provide supporting representation and attention analyses across additional medical modalities.

View source

Similar papers

Conference Open access Sep 2026

MedVCoT: Bridging the Modality Gap in Medical VQA Through Latent Visual Reasoning

This work proposes MedVCoT, which incorporates latent visual reasoning into the medical visual question answering (VQA) domain, and utilizes the specialized expertise of MedSAM to train a large vision-language model so that it can autonomously generate consistent and continuous latent visual tokens within Visual Chain-...

Bo Xu, Quan-Hao Zhu, Bo-Lin Zhu et al. · 1 citation
#artificial intelligence Preprint Sep 2026

EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding

EyeVQA is introduced, a unified visual question answering benchmark for comprehensive evaluation of ophthalmic VLMs, and fourteen representative general-purpose, scientific, and medically specialized VLMs under a unified zero-shot protocol are benchmarked.

Gu-Jie Shao, Zi-Xun Xie, Xue-Chun Xing et al. · 0 citations
#artificial intelligence Open access Oct 2026

Localization Lens for Improving Medical Vision-Language Models

Medical Vision-Language Models (Med-VLMs) have demonstrated strong capabilities in clinical tasks. However, they often struggle to understand anatomical structures and spatial positioning, which are crucial for medical reasoning. To address this, we propose a localization-aware enhancement to the Med-VLM pipeline, intr...

H. Farooq, Murtaza Taj, Mehwish Nasim et al. · 3 citations
#large language models Review Open access Sep 2026

Multimodal medical diagnosis: a mini review of LLM–vision fusion models in low-resource healthcare settings

Recent advances in large language models (LLMs) and vision transformers have enabled multimodal systems that integrate clinical text with medical imaging for diagnostic decision-making. While these systems show promising results on benchmark datasets in well-resourced research settings, their applicability in low-resou...

Kahakashan Ashraf, Md.Hamid Hosen, N. Farah et al. · 0 citations
Open access Aug 2026

Medrecord-CLIP: enhancing fundus disease diagnosis via EHR-guided vision-language pre-training

This work proposes MedRecord-CLIP, a knowledge-enhanced foundation model featuring a diagnosis-guided cross-attention mechanism to adaptively extract and fuse salient patient history with diagnostic representations that highlights the critical value of integrating personalized clinical context to enhance the generaliza...

Lei Shi, Wenbin Zhai, Lei Yu et al. · 0 citations
Preprint Sep 2026

A visual large language foundational model for medical image recognition using clinician-contributed online resources

Large language models (LLMs) have demonstrated strong capabilities across diverse domains, showing considerable potential in medicine. However, their application in medical settings remains limited by the scarcity of visual question answering (VQA) datasets that capture clinical reasoning and explicit image-text alignm...

Ling-Xuan Hou, Yu-Hua Xie, Yue Hu et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.