Skip to content

Answer or Defer? Testing VLMs'Stated Confidence on Brain MRI

Nov 2026 · 0 citations · 34 references
Computer Science

TL;DR

This work test whether thresholding stated confidence can retain useful coverage while limiting errors among accepted answers, and how this depends on the population, and separates confidence differences between predicted answers from differences within each answer.

Abstract

Vision-language models (VLMs) could support medical-image interpretation, but their use requires deciding which interpretations to accept and which images to refer to a clinician. We test whether thresholding stated confidence can retain useful coverage while limiting errors among accepted answers, and how this depends on the population. Eight open-weight VLMs, including two reasoning models, assess 3,888 brain MRI slices from post-treatment glioma patients (BraTS 2024) and healthy volunteers (IXI) for tumor or gross abnormality. BraTS labels are mask-derived; IXI slices are treated as negatives. On BraTS, no threshold selects a nonempty set with at most 5% empirical error for any model, even when chosen retrospectively. This also holds for four models evaluated on a large-lesion, distant-negative subset. At a 10% error budget, Gemma-4-12B retains 24.4% of slices and every other model at most 2.5%. To investigate this limited selection ability, we separate confidence differences between predicted answers from differences within each answer. For four models with available outcome-level summaries, pooled mean confidence differs more between answers than between correct and incorrect responses within each answer. A covariance decomposition explains how this pattern can produce opposite confidence-correctness associations across populations. At a 20% error budget, accepting every"tumor"answer covers more slices than any confidence threshold for two models. These findings caution against using unvalidated stated confidence as a standalone deferral signal and support population-specific validation before deployment. Code is available at https://github.com/amir-sbg/VLM-Answer-or-Defer.

View source

Similar papers

Conference Open access Oct 2026

When Reasoning Hurts Confidence: Demographic Calibration of Vision-Language Models in Medicine (Extended Abstract)

Vision-language models (VLMs) are increasingly evaluated as diagnostic aids in medical imaging, yet clinical deployment requires confidence estimates that are reliable and equitable across patient demographics. When a model reports 80% confidence, it should be correct about 80% of the time, for every patient group and...

Sonnet Xu, Roxana Daneshjou · 0 citations
#machine learning Preprint Nov 2026

Evaluating Cross-Signal Coherence in Mammography Vision-Language Models

As vision-language models (VLMs) move toward interpreting medical images directly, their outputs increasingly include not just a label but a stated confidence and a written justification. Yet the field still predominantly evaluates VLMs using just accuracy. We evaluate the coherence between answer, confidence, and expl...

Rebeca Nam, Chase Fensore, C. Fauria-Robinson et al. · 0 citations
Book Open access Nov 2026

MedVES: Visual Evidence Scaling at Test Time for VLMs in Medical VQA

Medical vision-language models (VLMs) hold significant promise for clinical image understanding, but scaling them at training time is limited by data privacy constraints and the cost and scarcity of expert annotations. Test-time scaling (TTS) is therefore a practical alternative, as it can improve performance without r...

Xiuzhe Wu, Hao-Ru Tan, Barathi Subramanian et al. · 0 citations
Preprint Nov 2026

Aggregate Accuracy Hides Diagnosis-Specific Failure in Medical Vision-Language Models

Medical vision-language models (VLMs) are compared mainly by accuracy averaged across findings, a summary that can hide complete failure on a single diagnosis. We audit five open medical VLMs (MedGemma-4B, Lingshu-7B, HuatuoGPT-Vision-7B, Qwen2.5-VL-7B, and MedVLM-R1) on 6,048 ChestX-ray14 questions balanced by sex, ag...

Shreya Ajay Kale, Angad Ripudaman Singh Bajwa · 0 citations
Open access Sep 2026

CARE-MVLM: Counterfactual Abstention and Region-Grounded Evidence in Mammography Vision-Language Models

CARE-MVLM substantially improves lesion-sensitive selective behavior over same VLM backbone baselines and exhibits markedly stronger lesion-sensitive abstention than GPT-5.6 and Gemini-3.5-flash while attaining the highest joint answer, grounding, and abstention reliability in the comparison.

B. Qu, W. Liu, M. Murrow et al. · 0 citations
Preprint Oct 2026

What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models

Vision--language models (VLMs) promise to make computational pathology queryable and able to draft reports, but how much language adds beyond the vision backbone is unclear, because almost every system is evaluated by its developers on test sets of their own choosing. We grade published evidence by how it was produced...

Wen-Hao Zhang, Zhong-Liang Zhou, Yi-Fang Ping et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.