Aug 2026· Frontiers in Artificial Intelligence· 0 citations· 16 references
TL;DR
Embedding domain-specific knowledge into LLMs may improve performance on specialized exam-style thoracic-surgery questions on this text-only benchmark, however, the present 56-item evaluation does not establish clinical equivalence, diagnostic accuracy in practice, multimodal competence, or readiness for real-world clinical decision support.
Abstract
Specialized thoracic-surgery questions require the integration of multi-factor clinical relationships within text, yet general-purpose large language models (LLMs) may underperform on such exam-style benchmarks.
We constructed a DK-LLM agent by embedding curated medical textbook knowledge into a LangChain-based framework to support domain-specific reasoning. The model was tested on a 56-item thoracic-surgery examination question set in a restricted text-only setting without internet browsing or external tools and was compared with generic LLM baselines, three thoracic surgeons, and three non-expert engineers. Examination score and error patterns were assessed.
The knowledge-augmented DK-LLM configuration showed an 11.8-point examination-score advantage over the version without the local knowledge base. Commercial LLM-based agents outperformed the open-source baselines and non-expert participants on this question set, whereas experienced thoracic surgeons achieved the highest scores overall; in the ablation analysis, removing the local knowledge base reduced the examination score by 11.8 percentage points.
Embedding domain-specific knowledge into LLMs may improve performance on specialized exam-style thoracic-surgery questions on this text-only benchmark. However, the present 56-item evaluation does not establish clinical equivalence, diagnostic accuracy in practice, multimodal competence, or readiness for real-world clinical decision support.
DDx-Finder is presented, an open-source framework that leverages Model Context Protocol (MCP) servers for direct EMR and literature access, enabling prompt-driven clinical state extraction and reliable case-report re- trieval via generating searching query by LLM, while addressing limitations related to resource demands and privacy concerns.
H. Lim, H. Yi, J. Y. Yoon et al.· medRxiv· 0 citations
The emergence of large language models offers unprecedented opportunities to transform clinical workflows. However, reliance on proprietary closed-source models poses significant risks to data privacy, institutional autonomy, and reasoning transparency. This study investigates the optimization of open-weight architectures, specifically the Llama-3.1-8B and Qwen2.5-14B-Instruct families, to function as specialized diagnostic tools rather than general-purpose conversational agents. Utilizing the MedQA USMLE-style benchmark, we evaluate the synergistic impact of context window scaling and multi-stage supervised fine-tuning within a localized Retrieval-Augmented Generation framework. Our findings identify a critical scaling threshold at 512 tokens, beyond which diagnostic accuracy stabilizes while preserving the computational efficiency required for on-site clinical deployment. Furthermore, we document a pronounced Brevity Shift, in which multi-stage supervised fine-tuning reduced model verbosity by over 99.7%. The models transitioned from long-form reasoning with an average of 465 tokens to single-token outputs for Qwen and near-single-token outputs for Llama. This shift effectively mitigated verbosity compensation, defined as the tendency of models to mask uncertainty through excessive text, while simultaneously improving diagnostic performance, which peaked at 70.0% accuracy for the Qwen architecture on the benchmark task. Additionally, our analysis reveals that systematic fine-tuning substantially reduces option-preference bias, as measured by Total Variation Distance, leading to more objective and consistent model outputs. Conducted under the INFOSTRATEG Strategic Program (PARROT AI) and funded by the National Centre for Research and Development (NCBR), Poland, this work establishes a robust technical framework for developing trustworthy, high-efficiency medical AI systems capable of operating entirely within hospital-controlled infrastructure.
L. Pawlik, Stanisław Deniziak· Scientific Reports· 0 citations
Background: Antimicrobial resistance poses a major threat to global public health. Large language models (LLMs) offer new possibilities for optimizing antibiotic prescribing decisions, but the capabilities of general-purpose versus domain-specific medical LLMs under different prompting strategies remain to be clarified. Methods: This double-blind, randomized-sequence evaluation used a 2X2 factorial design comparing four AI conditions-the domain-specific model MedGo and the general-purpose model DeepSeek V3.5, each under standard direct prompting and chain-of-thought (CoT) prompting-alongside real physician prescriptions across 59 complex inpatient infection cases. Five parallel regimens were generated per case and independently evaluated by three senior clinicians (1-5 comprehensive score and five domain sub-scores). ChatGPT 5.2 was additionally assessed as an automated evaluation tool. Results: Score ranking: real physicians > MedGo-CoT > DeepSeek-CoT > MedGo> DeepSeek (Friedman test, p<0.001). In base mode, MedGo significantly outperformed DeepSeek (Holm-adjusted p=0.040). CoT improved both models (Holm-adjusted p<0.001 for DeepSeek; p=0.024 for MedGo) and reduced score dispersion. MedGo-CoT significantly outperformed DeepSeek-CoT in individualized adjustment (adjusted p<0.001) and dosing precision (adjusted p=0.005). ChatGPT-expert correlation was negligible (overall Kendall {tau}=0.153, p=0.003; subgroup {tau}=0.06-0.20, all p>0.05). Conclusions: Domain-specific medical LLMs enhanced by CoT approach the antibiotic decision-making level of real physicians, with advantages in individualization and dosing precision. However, notable deficiencies persist in antimicrobial stewardship ecological awareness and automated evaluation reliability, underscoring the continued indispensability of senior clinical expertise.
Y. Liu, C. Zhang, F. Wang et al.· medRxiv· 0 citations
Findings show that clinical LLM explainability has shifted toward fluent generative rationales, but evidence that such explanations reflect model reasoning remains limited, and three regulatory priorities are highlighted: prioritizing explanations that enable independent verification or logic auditing over plausibility-only rationales; preferring inspectable models where regulatory documentation is required; and prospectively validating explanations in clinical workflows before scaling.
: High-stakes clinical decision support (CDS) demands a property that aggregate accuracy cannot capture: a trace that a clinician who was not in the room can inspect layer by layer when the system is wrong. We argue that the way to obtain this property is to refuse to entangle the large language model (LLM) with the rest of the pipeline. We propose KRD (Knowledge–Rule–Decision) , a four-component architecture that separates fact extraction, a compile-time clinical knowledge layer in the spirit of the LLM Wiki pattern of Karpathy, a rule layer of hand-written contraindications and heuristics, and a decision interface whose compose method short-circuits to a rule-cited blocking response whenever any hard violation fires. We evaluate KRD against a pure language model, a retrieval-augmented language model, a rule-only system, and a light hybrid on a benchmark of 32 type-1 diabetes scenarios. A strict version of the unsafe-suggestion rate stratifies the five systems monotonically into four distinct tiers from 0.867 down to zero, with S4 and S5 tied at the floor; the full KRD stack and the light hybrid reach the hard-safety ceiling together; KRD leads the light hybrid on evidence trace completeness by 25% relative and on reviewer correction burden by 12% relative, both directionally clear and borderline significant under bootstrap intervals; and KRD issues 17 language model calls per benchmark pass against the light hybrid’s 32, a 47% reduction that is a direct consequence of the architectural choice to evaluate the rule layer before invoking the model. We also report honestly that the evidence gate is inert on this benchmark because every compiled concept is graded A or B, and we trace five fact-extraction failures to a single field and a single linguistic pattern. The contribution is not that KRD is universally optimal but that layer-wise auditability is a design discipline whose cost in this setting was lower than its critics would have predicted.
Bailing Zhang, Genlang Chen· Journal of Intelligent Medic...· 0 citations
A single LLM verifier lacks sufficient reliability to serve as a stand-alone judge of clinical reasoning at scale, and that structured human oversight remains essential.
Hyunjung Byun, Dahyoun Lee, Munyoung Jung et al.· Journal of medical systems· 1 citation· ⚡1