Few-Shot Spatial Adaptation in Medical Vision-Language Models
Vision-language models (VLMs) have significantly advanced open-vocabulary image understanding by learning aligned representations from large-scale image-text datasets. Despite their zero-shot generalization capabilities, adapting these foundation models to specific application domains remains challenging. Full fine-tuning is often infeasible due to computational costs and the risk of overfitting when labeled data are limited. Parameter-efficient fine-tuning (PEFT) approaches promise to address this issue by updating only a small set of parameters while keeping the pre-trained encoders largely frozen. However, existing PEFT strategies often exhibit insufficient spatial awareness, rendering them suboptimal for medical imaging, where subtle visual differences can lead to distinct clinical diagnoses. To address this challenge, in this study we propose a novel PEFT technique that leverages natural-image pretrained DINOv3’s attention maps to enforce spatial alignment.