MedVES: Visual Evidence Scaling at Test Time for VLMs in Medical VQA
Abstract
Medical vision-language models (VLMs) hold significant promise for clinical image understanding, but scaling them at training time is limited by data privacy constraints and the cost and scarcity of expert annotations. Test-time scaling (TTS) is therefore a practical alternative, as it can improve performance without retraining. However, existing TTS methods are designed mainly for LLMs, where extra inference compute is spent on language-space reasoning. This paradigm does not transfer directly to medical visual reasoning, where generating more text does not yield new visual findings. Instead, progress requires iteratively seeking and integrating fine-grained, localized visual evidence from different image regions. We introduce MedVES, a training-free framework for visual evidence scaling at test time for frozen VLMs in medical VQA. MedVES casts test-time scaling as iterative visual evidence seeking rather than extended language-space reasoning. Its core component is a Visual Overlay State (VOS), a bounded visual state representation that externalizes accumulated evidence on the original image while preserving global context. Built on VOS, MedVES performs iterative visual evidence acquisition with adaptive stopping, enabling test-time compute to scale with question complexity. Across five medical VQA benchmarks covering diverse imaging modalities, a 4B MedVES model achieves strong average performance at its scale and is competitive with, or better than, several models with 8× more parameters, without any additional training. Code is available.