Large language models generate diagnostic likelihood ratios with low mean bias but wide dispersion.
TL;DR
It is indicated that LLM-generated LRs can be centered near literature values on average, but they are not interchangeable with published estimates for individual high-stakes decisions.
Abstract
Accurate, context-appropriate likelihood ratios (LRs) are needed for Bayesian diagnosis, but empirical LRs are sparse because diagnostic accuracy studies are costly and context-dependent. We evaluated whether large language models (LLMs) can generate LR outputs that align with literature-reported values. We compared LR outputs generated by three OpenAI models (GPT-4o, o3, and GPT-5) with all literature-reported values curated in TheNNT.com dataset. A few-shot prompt was used to elicit numerical LR outputs, and agreement was assessed using Bland-Altman analyses to evaluate mean bias and multiplicative limits of agreement. A total of 700 reported LRs across 30 conditions were compiled, most involving signs or symptoms (59%), historical elements (19%), or test results (16%). All models demonstrated negligible mean bias. GPT-5 had the narrowest 95% limits of agreement (0.26×-3.70×) compared with o3 and GPT-4o. These findings indicate that LLM-generated LRs can be centered near literature values on average, but they are not interchangeable with published estimates for individual high-stakes decisions. Clinical use, including use for unstudied findings, would require prospective, context-specific diagnostic-accuracy validation.