Cross-modal bias in medical vision-language models: a pipeline-aware framework for mechanisms, evaluation, and mitigation

Abstract

Medical vision-language models encode images and clinical text in a shared representation. Across radiology and ophthalmology, their diagnostic performance now approaches that of specialist clinicians. The mechanism behind that performance is also the source of a problem that has gone largely unexamined. These models are trained by contrastive alignment, so bias from the image encoder and bias from the text encoder meet at a single point: the alignment interface. There they can interact and compound in ways that single-modality systems never experience. Fairness research has so far studied the two modalities separately, and medical vision-language models have fallen into the gap between those literatures. We organize the evidence into a three-tier taxonomy keyed to the pretraining pipeline. Tier 1 is data-level bias in the pretraining corpus. Tier 2 is alignment bias produced at the contrastive interface. Tier 3 is inference-time bias that surfaces during deployment. Within this structure, we compare the major evaluation benchmarks, show where they disagree, and assign each mitigation strategy to the tier it actually addresses. Three points emerge. First, no published method spans all three tiers; mitigation is fragmented by pipeline stage. Second, fine-tuning does not remove alignment-stage bias, even in parameter-efficient form, which shifts the burden of debiasing onto pretraining rather than adaptation. Third, the inference-time failures are more dangerous than the literature suggests. Medical-specialist models will abandon a correct reading and defer to a confident user on most trials, and specialization appears to make this worse, not better. We close with a research agenda.