When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning
This work systematically examines how MedSAM generalizes across diverse medical imaging benchmarks, with six adaptation strategies: full-model and encoder-only LoRA, shallow and deep visual prompt tuning (VPT), and decoder-only and full fine-tuning, and concludes that robust MedSAM adaptation requires the combined cons...