Jun 2026· arXiv.org· Vol abs/2606.29034· 1 citation· 57 references
Computer Science
TL;DR
Clinical LLMs carry an ordered evidence-strength signal they do not express, so their stated grades fail to convey a claim's support even when it is recoverable from their representations and text.
Abstract
Large language models (LLMs) increasingly summarize clinical evidence, where a claim's weight depends on how strongly it is supported. Yet these models convey confidence poorly, and properties they never state, such as truth, are often readable from their activations. Whether a clinical model registers evidence strength, distinct from truth, and states it when asked is untested, and any such signal could be lexical. We compiled 45,134 clinical claims from six public sources, harmonized 20,611 into a four-level evidence grade under three independent frameworks, and tested 22 local, open-weight LLMs from several developers (0.6-70 billion parameters; general, medical, and reasoning), with lexical, truth, and cross-framework controls. A linear estimator recovered the grade in every model (median AUROC 71.8), yet decodability did not rise with scale and was weakest in reasoning models. The grade the models stated fell to chance, 25-27 percentage points below the estimator. The recoverable signal was largely lexical and did not transfer across topics or frameworks, yet it was distinct from factual truth and still flagged weakly supported claims (AUROC 69.2). Clinical LLMs thus carry an ordered evidence-strength signal they do not express, so their stated grades fail to convey a claim's support even when it is recoverable from their representations and text.
Findings show that clinical LLM explainability has shifted toward fluent generative rationales, but evidence that such explanations reflect model reasoning remains limited, and three regulatory priorities are highlighted: prioritizing explanations that enable independent verification or logic auditing over plausibility-only rationales; preferring inspectable models where regulatory documentation is required; and prospectively validating explanations in clinical workflows before scaling.
Large Language Models (LLMs) achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably. A central risk is an evaluation illusion: fluent and well-structured explanations can appear clinically convincing even when the final diagnosis is incorrect. We introduce CLExEval, a human-in-the-loop framework for evaluating LLM clinical reasoning under progressive information masking. CLExEval combines 5,600 expert-physician annotations with 200 clinical reasoning traces derived from 40 rare diagnostic cases. Our analysis identifies three recurring failure patterns: (i) verbosity bias, where GPT-4o-mini's diagnostic accuracy drops from 95.0% to 32.5% under information scarcity; (ii) a hidden knowledge paradox, where a specialist model reaches 92.5% maximum diagnostic potential but fails to retrieve that knowledge reliably in verbose contexts; and (iii) a 68.6% reasoning-to-output mismatch, where correct diagnoses appear in reasoning traces but are not reflected in final answers. We further evaluate the LLM-as-a-Judge paradigm on a human-verified failure set (n = 142). GPT-4o-mini approved 47.9% of clinically incorrect outputs, while HuatuoGPT-o1 approved all validly scored failures and showed a positive self-preference bias. These results suggest that standalone automated clinical evaluations can substantially overestimate clinical reliability without expert-grounded validation.
Abin Roy, Afthab Salam Kanniyan, Jawadh Abdul Kabeer et al.· arXiv.org· 0 citations
Whether clinical safety established in English transfers to Hausa is asked, and whether any failure is attributable to the language, the clinical task, or the class of model that low-resource deployment admits.
Anthonio Oladimeji Gabriel, Dimeji AbdulSobur Olawuyi, T. Ajayi et al.· 0 citations
This paper argues that longitudinal clinical reasoning is a state-estimation problem under partial observability, and that the axis on which clinical AI succeeds or fails is not the fluency of the model reading the record but the governance of the patient state it reasons over.
Augusto Bernardo Pissarra, Victor Farias DE Souza· 0 citations
Large language models achieve high scores on medical knowledge assessments, yet clinical reasoning requires actively deciding what to investigate under uncertainty. We developed an agentic evaluation framework in hematologic oncology in which models must proactively request clinical data across three sequential rounds before committing to a diagnosis and treatment plan. Across 32 frontier models, the best achieved only 68% overall accuracy. Information utilization, the fraction of available data actually requested, was the strongest predictor of diagnostic accuracy (R = 0.69, P<0.001), yet utilization collapsed from 57% to 26% in the final round, leaving molecular and cytogenetic data critical for treatment selection unexamined. Reasoning traces scored high on a clinical reasoning rubric (91% above threshold) but decorrelated from accuracy, revealing a gap between locally coherent rationales and globally correct conclusions. Error analysis identified search satisficing, anchoring and premature closure as the dominant failure modes, the same cognitive biases that characterize novice clinicians under dual-process models of diagnostic reasoning. These findings demonstrate that the primary limitation of current models in clinical oncology is not insufficient medical knowledge but a systematic failure of information-seeking under uncertainty.
K. Braitsch, L. Schmalbrock, Theresa Weltermann et al.· 1 citation
Clinical deployment of large language models (LLMs) for medical question answering faces three primary challenges: (1) models frequently produce unsupported or hallucinated claims because their parametric knowledge is not verified against biomedical evidence; (2) single-pass inference prevents iterative self-correction, leading to incomplete differential diagnoses; and (3) poorly calibrated confidence scores undermine clinician trust. To address these challenges, we introduce a retrieval augmented multi-agent model whose components are specifically designed to generate medical answers through three stages: hypothesis generation, adversarial debate, and independent judgment, wherein evidence conditioned hypotheses are generated, contested by an adversarial process, and finally evaluated by an independent model. Each of these models utilizes a semantic retrieval system to access medical literature from PubMed, explanations of answer benchmarks from MedQA, and medical records from MIMIC III. The model is evaluated using the MedQA USMLE benchmark and a collection of prompts created from MIMIC III data, as well as by comparing the performance of the system against a general purpose LLM and a fine tuned medical LLM. Results indicate that the approach described improves the supported sentences rate, the precision and recall of the attributed sentences, and the groundedness of the answers provided while also reducing the number of unsupported claims, hallucinations, Brier score, and expected calibration error, while maintaining the same accuracy in medical diagnosis. Thus, these findings indicate that the incorporation of adversarial and verification models during inference can improve the model’s ability to align with the evidence and to calibrate uncertainty in its predictions without requiring any adaptation to the models with supervised training data.
Shreyas Bhoyar, Adil Ansari, N. Saxena· 2026 IEEE International Conf...· 0 citations
