The findings suggest that engineering diagnostic AI as a safety-oriented reasoning framework, rather than optimizing raw predictive accuracy alone, can provide a safer, more transparent, and clinically meaningful layer of bedside decision support for acute care workflows.
Abstract
Diagnostic error is a major threat to patient safety, yet current large language model (LLM) systems often treat diagnosis as a one-shot prediction task, lacking safeguards against missed high-risk alternatives or rigorous verification of their reasoning. Here, we present AegisDx, a safety-oriented framework for hypothetico-deductive clinical reasoning. AegisDx coordinates specialized LLM components through role-specific contracts, structured intermediate outputs, evidence-retrieval interfaces, and verification gates to generate broad differential diagnoses, enforce explicit screening for dangerous"must-not-miss"conditions, verify reasoning against grounded medical evidence, and structure actionable next steps. We evaluated AegisDx across three layers. On literature-derived case reports from NEJM and JAMA, with GPT-oss-120B as the shared backbone, Top-3 diagnostic accuracy was 59.9% versus 52.1% for the standalone LLM on JAMA cases and 62.7% versus 51.4% on NEJM cases. On cases from Annals of Emergency Medicine, Top-3 accuracy was 85.7% versus 68.6%; against physician-consensus must-not-miss diagnosis sets, AegisDx captured at least one such condition among its top three diagnoses in 78.0% of cases versus 52.0%. In a blinded physician evaluation of 43 real-world emergency department notes from the Yale New Haven Health System compared against GPT-5, AegisDx improved the physician-rated composite safety score from 4.31 to 4.55 on a 5-point scale (adjusted p = 2.1x10^-4), with qualitative gains in must-not-miss identification and reasoning safety. Our findings suggest that engineering diagnostic AI as a safety-oriented reasoning framework, rather than optimizing raw predictive accuracy alone, can provide a safer, more transparent, and clinically meaningful layer of bedside decision support for acute care workflows.
SymDiag is proposed, a neuro-symbolic framework that reframes reasoning verification as structured failure diagnosis and incorporates a Self-Auditor that disentangles TranslationError from ReasoningError via dual symbolic encodings consistency checks, enabling robust diagnosis under partial observability.
Wenyao Cui, Huaping Zhang, Yongyi Huang et al.· Proceedings of the 32nd ACM...· 0 citations
Large language models (LLMs) often rely on shortcuts rather than systematic reasoning, raising safety concerns in medical applications. Allowing models to abstain when uncertain improves reliability but introduces a coverage accuracy tradeoff. We propose a two-stage framework for medical hypothesis verification in multiple-choice settings that manages this tradeoff through targeted ontology grounding, applied only when the model abstains. We show that abstention is not random but reflects genuine uncertainty, with abstained predictions associated with lower confidence. Across two frontier models (GPT-5.5, accessed via the Azure OpenAI API, and DeepSeek-R1), the proposed framework improves question-level accuracy by 9.6 percentage points (82.9% to 92.5%) and hypothesis-level accuracy by 4.2 percentage points (92.0% to 96.2%). Our experiments conducted on MedReason and MedQA show that abstention can be repurposed as a control signal for selective reasoning refinement, achieving knowledge-graph-level performance without explicit knowledge graph construction.
Uma Ranjan, Kunal Tilaganji, Aditya Koul et al.· 0 citations
Clinical artificial-intelligence systems are starting to act across a course of care, not to answer one question at a time. Their safety is checked by methods that sample the input space: a test suite tries some inputs, a language-model reviewer reads some cases, a physician panel audits some cases. A sampling check can pass a safety rule and still miss the rare input that breaks it, such as a documented obligation dropped several encounters later. This study measures that gap and releases CIV-Bench, a public benchmark of 832 clinical rule sets with safety properties across eight whole-person domains, in single-encounter and longitudinal forms, plus a computational stress tier, each with independently established ground truth. We compare formal verification, which uses a satisfiability-modulo-theories (SMT) solver to check every possible input at once, against the methods used in practice: random unit testing, language-model judges, and a blinded physician panel. Formal verification detected all 612 violations, raised no false alarm, and returned no unsound verdict; for each item it returned either a proof that the rule holds over every input or one concrete input that breaks it. A frontier language-model judge matched this detection, but it returned a pass rate over sampled cases rather than a guarantee, at three orders of magnitude more compute per item. The general open-weights judge returned unsound verdicts on the computational stress tier; the medically fine-tuned judge was unsound far more widely, collapsing on the longitudinal properties despite strong single-encounter medical detection, so medical fine-tuning did not close the gap. Unit testing and the physician panel missed the deep, cross-encounter violations that hold a course of care together. Formal verification is set apart not by a higher detection rate but by the kind of evidence it returns: a proof over the whole input space, a replayable counterexample, or an explicit statement that it cannot decide. The guarantee holds for the decisions placed in this layer, and it depends on the safety rule being specified correctly.
Large language models achieve high scores on medical knowledge assessments, yet clinical reasoning requires actively deciding what to investigate under uncertainty. We developed an agentic evaluation framework in hematologic oncology in which models must proactively request clinical data across three sequential rounds before committing to a diagnosis and treatment plan. Across 32 frontier models, the best achieved only 68% overall accuracy. Information utilization, the fraction of available data actually requested, was the strongest predictor of diagnostic accuracy (R = 0.69, P<0.001), yet utilization collapsed from 57% to 26% in the final round, leaving molecular and cytogenetic data critical for treatment selection unexamined. Reasoning traces scored high on a clinical reasoning rubric (91% above threshold) but decorrelated from accuracy, revealing a gap between locally coherent rationales and globally correct conclusions. Error analysis identified search satisficing, anchoring and premature closure as the dominant failure modes, the same cognitive biases that characterize novice clinicians under dual-process models of diagnostic reasoning. These findings demonstrate that the primary limitation of current models in clinical oncology is not insufficient medical knowledge but a systematic failure of information-seeking under uncertainty.
K. Braitsch, L. Schmalbrock, Theresa Weltermann et al.· 1 citation
This Perspective concerns the most consequential of these applications: the autonomous triage of self-presenting, undifferentiated patients, with little or no clinician in the loop, and the evidence of safety does not yet exist.
Shayndhan Sivanathan, Shravan Nageswaran, Mehdi Zadem et al.· 0 citations
Large Language Models (LLMs) show strong potential for clinical reasoning, yet their deployment in medical decision support is hindered by hallucinations, overconfidence, and limited transparency. We propose Dialectic Diagnosis, an agentic framework in which two heterogeneous LLM agents engage in structured argumentative interaction inspired by clinical second-opinion workflows. A Clinical Reasoner proposes candidate diagnoses, while a Skeptical Critic challenges these hypotheses by identifying omissions, cognitive biases, and unsupported reasoning. Their interaction is governed by a formal finite-state machine (FSM) enforcing a disciplined proposecritique-resolve protocol, with final decisions produced by an Arbiter agent providing calibrated confidence estimates. To ensure transparency, we introduce a Diagnostic Argument Graph that explicitly represents supporting evidence, contradictions, and missing diagnoses. Evaluations on real-world clinical datasets (MIMIC-IV and eICU) show clear gains over single-agent LLM baselines, with improved diagnostic accuracy, lower calibration error, and fewer critical diagnostic omissions. These results indicate that structured argumentative interaction between LLM agents provides a principled path toward safer and more explainable clinical AI systems.
Belkacem Chikhaoui· Annual International Compute...· 0 citations