How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective
A mechanistic analysis of the VLM model's internal attention patterns shows that a simple describe-then-decide prompting strategy increases vision attention by 30-40% during generation, and task-specific fine-tuning improves dermatology classification but reduces cross-domain medical question-answering performance in t...
Janet Wang, Yun-Bei Zhang, Xiao Wang et al.
· 0 citations