Enhancing clinical reasoning and diagnostic precision through scaling laws and multi-stage supervised fine-tuning in open-weight medical large language models
TL;DR
A robust technical framework is established for developing trustworthy, high-efficiency medical AI systems capable of operating entirely within hospital-controlled infrastructure by evaluating the synergistic impact of context window scaling and multi-stage supervised fine-tuning within a localized Retrieval-Augmented Generation framework.
Abstract
The emergence of large language models offers unprecedented opportunities to transform clinical workflows. However, reliance on proprietary closed-source models poses significant risks to data privacy, institutional autonomy, and reasoning transparency. This study investigates the optimization of open-weight architectures, specifically the Llama-3.1-8B and Qwen2.5-14B-Instruct families, to function as specialized diagnostic tools rather than general-purpose conversational agents. Utilizing the MedQA USMLE-style benchmark, we evaluate the synergistic impact of context window scaling and multi-stage supervised fine-tuning within a localized Retrieval-Augmented Generation framework. Our findings identify a critical scaling threshold at 512 tokens, beyond which diagnostic accuracy stabilizes while preserving the computational efficiency required for on-site clinical deployment. Furthermore, we document a pronounced Brevity Shift, in which multi-stage supervised fine-tuning reduced model verbosity by over 99.7%. The models transitioned from long-form reasoning with an average of 465 tokens to single-token outputs for Qwen and near-single-token outputs for Llama. This shift effectively mitigated verbosity compensation, defined as the tendency of models to mask uncertainty through excessive text, while simultaneously improving diagnostic performance, which peaked at 70.0% accuracy for the Qwen architecture on the benchmark task. Additionally, our analysis reveals that systematic fine-tuning substantially reduces option-preference bias, as measured by Total Variation Distance, leading to more objective and consistent model outputs. Conducted under the INFOSTRATEG Strategic Program (PARROT AI) and funded by the National Centre for Research and Development (NCBR), Poland, this work establishes a robust technical framework for developing trustworthy, high-efficiency medical AI systems capable of operating entirely within hospital-controlled infrastructure.