Supplemental Material for: Performance of large language models combined with clinical decision support for recurrent pregnancy loss management

Abstract

Background: Recurrent pregnancy loss (RPL) affects 1%–5% of women worldwide. Advanced maternal age and increasing pre-pregnancy coexisting diseases have complicated its clinical management. Large language models (LLMs) and clinical decision support (CDS) tools are widely used clinically, yet combined use evidence for RPL remains scarce. Objective: To evaluate whether structured CDS materials improve the guideline compliance of three LLMs for standardized RPL clinical scenarios. Design: Comparative simulation study with self-controlled two-stage assessment. Method: Two reproductive medicine specialists developed 25 standard clinical scenarios based on the 2018 and 2022 ESHRE guidelines. Three LLMs (ChatGPT-5.5, Claude-4.8, DeepSeek-R1) were assessed in two phases: baseline evaluation and assessment with structured CDS figures. Response compliance with guidelines was evaluated. Weighted pooled Cohen’s kappa was used to assess inter-rater reliability. The Friedman test with Bonferroni correction was applied for inter-group comparisons, and the Wilcoxon signed-rank test for paired analyses. Results: At baseline, the full guideline compliance rates of the ChatGPT-5.5, Claude-4.8, and DeepSeek-R1 were 44.0%, 28.0%, and 20.0%, with significant differences among groups (P<0.05). After CDS intervention, the rates increased to 92.0%, 84.0%, and 80.0%. All models showed statistically significant improvements (ChatGPT-5.5: P=0.00033; Claude-4.8: P=8.2×10⁻⁵; DeepSeek-R1: P=2.8×10⁻⁵). Dimension-based analysis revealed that all three models achieved a 100% accuracy rate in judging the definition of RPL throughout the study. With the support of CDS tools, their performance in core items, including key screening, etiological, recommended, and non-recommended interventions, was greatly enhanced. No seriously incorrect responses were observed in the whole process. Limitations: This study only included 25 typical clinical scenarios, which is a relatively small sample. In addition, all CDS reference materials were presented in a single unified table format. Conclusion: LLMs only deliver moderate performance in baseline RPL management and are not competent to replace clinical judgment. Combining LLMs with structured CDS tools can remarkably improve the consistency of their outputs with clinical guidelines. This combination can serve as a reliable auxiliary tool for routine RPL clinical work.