Safety in the Age of Artificial Intelligence: Evaluating Large Language Model Adherence to Antithrombotic Medication and Regional Anesthesia Guidelines
The release of the 2025 American Society of Regional Anesthesia (ASRA) 5th edition guidelines for regional and neuraxial anesthesia procedures for patients receiving antithrombotic medications introduced complex, patient-specific hold times and resumption protocols. As clinicians increasingly utilize large language models (LLMs) as clinical decision support tools, the reliability of these models remains largely unvalidated. This study evaluates the accuracy of two of the foremost LLMs, ChatGPT (OpenAI, San Francisco, CA) and Google Gemini (Google DeepMind, London, UK), in adhering to these new gold-standard safety guidelines. Twenty-five standardized clinical vignettes were developed. Each vignette featured a patient on a specific anticoagulant (e.g., rivaroxaban, apixaban, dabigatran) requiring a neuraxial or regional anesthetic procedure (stratified by high-risk vs. low-risk). Variables included renal function, dose frequency, and procedural urgency. Prompts were submitted to the latest publicly available ChatGPT and Google Gemini models with separate instructions to provide hold and resumption times. LLMs were queried to ensure familiarity with 2025 ASRA guidelines prior to submission of prompts. Responses were graded against the 2025 ASRA guidelines by independent reviewers. Response adherence was categorized as: 1. concordant (100% match); 2. conservative error (LLM recommended time was longer than required); 3. dangerous error (recommended time was shorter than required, a critical safety violation); or 4. omission (no specific timeframe provided). ChatGPT achieved a concordance rate of 64%, compared to Gemini at 62%. However, the models displayed distinct error profiles. ChatGPT produced "dangerous errors" in 20% of evaluations and failed to specify a time in 16% of cases. In contrast, Gemini’s dangerous error rate was lower at 12%, but it demonstrated a significant "conservative error" rate of 22%, compared to 0% for ChatGPT. A chi-square test indicated that the difference in dangerous error rates between the two models was not statistically significant. Regardless, notable differences in error character and response completeness were observed. Gemini was more consistent in providing specific timeframes in 96% of prompts compared to 84% for ChatGPT. Variability of responses between users was determined via a chi-square test of homogeneity and was found to be statistically significant only for Gemini. While both models demonstrated moderate guideline awareness, their failure modes differed meaningfully. ChatGPT's errors were predominantly dangerous underestimations and omissions, while Gemini exhibited a conservative bias, overestimating hold times when a match was not achieved. Significant limitations exist regarding generalizability of these results. Only two major LLM models were tested. Other, more clinically oriented models exist, and newer versions of both ChatGPT and Gemini are consistently being released. These may improve LLM adherence to clinical guidelines. Despite these limitations, this study highlights the dangers of utilizing LLMs for periprocedural anticoagulation decision-making in regional and neuraxial anesthesia. Both models produced potentially dangerous recommendations in 12-20% of scenarios. Ongoing evaluation of LLM adherence to evolving guidelines remains essential, and clinicians must exercise extreme caution when utilizing these tools in patient care.