Skip to content

Can LLMs Value the Right Evidence? Evidence-Value Misalignment in Dynamic Medical Diagnosis

Sep 2026 · 0 citations · 35 references
Computer Science

TL;DR

It is demonstrated that verifying evidential support before submission can make diagnostic decisions more reliable, and decouples diagnosis generation from submission through an offline Contrastive Diagnostic Wiki and three online control stages, namely observation management, proposal and witness verification, and diagnosis submission control.

Abstract

A correct diagnosis reached from insufficient or misleading evidence can pose a clinical hazard, yet outcome-based accuracy may reward such lucky guesses. We call this mismatch between diagnostic decisions and the value of available evidence Evidence-Value Misalignment (EVM). To disentangle evidential grounding independently from diagnostic accuracy, we introduce MedEVM, a dynamic benchmarking environment comprising 1,050 cases across 24 disease systems. Observations arrive turn by turn, requiring models to continuously calibrate its decision by deciding whether to wait for more evidence or submit a diagnosis. Across 9 LLMs, four interesting patterns are observed. (1) Miscalibrated evidence tracking. Making a diagnosis often fails to calibrate evidence sufficiency, even in more capable models, and even worsens in reasoning mode. (2) Misaligned diagnosis submission. Confidence in the correct diagnosis often fails to ensure timely submission despite sufficient evidence. (3) Evidence order matters. Reordering the same evidence changes diagnoses even when model confidence remains similar. (4) Misleading evidence remains influential. Added misleading evidence redirects diagnoses even after prior evidence becomes sufficient. We further verify that EVM predicts errors and that preventing premature submission improves accuracy. These findings motivate Evidence-Verified Diagnosis Harness (EVD-Harness). It decouples diagnosis generation from submission through an offline Contrastive Diagnostic Wiki and three online control stages, namely observation management, proposal and witness verification, and diagnosis submission control. Across five LLMs, EVD-Harness improves accuracy by 12.0--51.1 percentage points while mitigating EVM-related failures. Our results demonstrate that verifying evidential support before submission can make diagnostic decisions more reliable.

View source

Similar papers

Preprint Aug 2026

CDEG: Learning Decision-Critical Evidence for Long-Horizon Diagnostic Agents

CDEG, a graph-based framework that learns reusable decision-critical evidence from historical diagnostic trajectories, is introduced, demonstrating that reliable long-horizon diagnosis requires moving beyond trajectory-level experience reuse toward evidence-level learning of the factors that truly shape clinical decisi...

Xi-Wei Dai, Zi-Jie Meng, Zhiting Fan et al. · 0 citations
#artificial intelligence Preprint Aug 2026

From Uncertainty to Clinical Risk: Severity-Aware Conformal Planning for Interactive Medical Diagnosis

S Severity-Aware Conformal Clinical Planning formulates interactive diagnosis as a risk-sensitive sequential decision problem and achieves more accurate diagnoses with fewer questions across multiple large language models, while improving differential-diagnosis quality and reducing high-risk errors in severe cases.

Yue Zhou, Haisu Zhou, Jin Zhang et al. · 0 citations
Preprint Aug 2026

EviDx: Evidence-Aware Active Diagnosis with Scaffolded LLM Agents

EviDx is introduced, an evidence-aware active diagnosis framework that pairs patient-specific diagnostic environments with a clinical diagnostic scaffold and an observer-guided runtime harness that improves diagnostic performance and process stability while revealing model-dependent capability boundaries.

Lihang Zeng, Shao-Ting Zhang, Xiao-Fan Zhang · 2 citations
#artificial intelligence Preprint Sep 2026

Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning

Longitudinal clinical reasoning requires large language models (LLMs) to identify and integrate relevant evidence distributed across extended patient histories. Although long-context models can process increasingly large amounts of information, providing more history does not necessarily make relevant evidence more acc...

Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.