Large language models reveal a systematic preference in clinical immunotherapy guidelines: a multi-model comparative analysis

Abstract

Clinical guidelines may sometimes contain divergent treatment recommendations, creating uncertainty in therapeutic decision-making. To evaluate the performance of four large language models (LLMs) (GPT-4o, Claude 3.7 Sonnet, Gemini 1.5 Pro, Deepseek-V3) in analyzing discrepancies between the American Society of Clinical Oncology (ASCO) and the National Comprehensive Cancer Network (NCCN) immune checkpoint inhibitors (ICIs) guidelines. ICI-related guidelines published by ASCO and NCCN were collected, divergent recommendations were extracted and input into four LLMs for statistical analysis, and each model’s responses were independently evaluated by three oncology experts using Likert scales (focusing on accuracy, comprehensiveness, readability, and potential harm). All models demonstrated a significant preference for NCCN. Gemini 1.5 Pro exhibited extreme rigidity (100% selection of NCCN). The remaining models showed moderate degrees of a systematic preference: Claude 3.7 Sonnet(73.7%) , Deepseek-V3(72.0%), and GPT-4o(71.3%) . Further stratification revealed a relative preference for ASCO guidelines under more conservative treatment protocols. Generative artificial intelligence (AI) demonstrates persistent and systematic preference in conflicting guideline interpretation, with an overall inclination toward NCCN guidelines, potentially related to training corpus distribution and differences in guideline articulation. These findings suggest the need to establish transparent and regulatory-compliant AI decision-making frameworks.