Evaluating the Accuracy of ChatGPT-4o in Addressing Complex Clinical Questions Based on NCCN Guidelines for Rectal Adenocarcinoma.

ChatGPT-4o as a decision-support tool in a urological tumour board: a prospective evaluation.

Final recommendation concordance did not meet the protocol-defined benchmark, ChatGPT-4o never altered an MTB decision, and clinically relevant errors occurred even among highly concordant outputs, showing that concordance alone does not guarantee safety.

Guideline Concordance of Large Language Models in the Management of Ureteral Stones: A Clinical Vignette-Based Comparative Study

ObjectiveLarge language models (LLMs) are increasingly used to answer medical questions, but their reliability in guideline-based urological decisions remains uncertain. This study aimed to evaluate the concordance of three widely used LLMs with the European Association of Urology (EAU) 2026 guideline recommendations for the active management of ureteral stones.Materials and MethodsIn this cross-sectional, vignette-based comparative study, the EAU 2026 ureteral-stone treatment algorithm was converted into 40 standardized clinical vignettes (four groups of ten: proximal 10 mm, distal 10 mm). ChatGPT, Gemini, and Claude were queried with the same standardized prompt, which did not name a specific guideline. Responses were scored against predefined EAU-based reference answers using a binary system. Concordance was compared with Cochran’s Q test.ResultsA total of 120 LLM-generated responses were evaluated. Overall concordance was 96.7% (116/120). ChatGPT achieved complete concordance (40/40, 100%), while Gemini and Claude each achieved 95% (38/40); the difference was not significant (Cochran’s Q=2.67, p=0.264). Concordance was complete in all 10 mm scenarios requiring URS prioritization. All four discordances were “incorrect prioritization” in >10 mm stones, presenting shock-wave lithotripsy as co-equal to ureteroscopy; each involved cross-guideline conflation with American Urological Association (AUA) framing. No unsafe recommendation or guideline hallucination was observed under the predefined scoring categories.ConclusionThe evaluated LLMs showed high concordance with EAU 2026 first-line treatment recommendations for ureteral stones. Concordance was complete in non–priority-sensitive

Performance of leading large language models in adhering to clinical guidelines for anaplastic thyroid cancer: a comparative study

Leading LLMs show variable capacity to align with ATC clinical guidelines, while top-performing models hold promise as supportive tools, their inconsistencies across domains and complexity levels preclude autonomous clinical use.

Guideline-based evaluation of large language models for patient counselling before robot-assisted radical prostatectomy: accuracy, safety, and readability

DSAI-12 ACCURACY, SAFETY, AND READABILITY OF PUBLIC-FACING LARGE LANGUAGE MODELS IN CNS METASTASIS

Although contemporary LLMs increasingly reflect medical consensus for CNS metastases, inconsistent reliability remains a concern, underscoring the need for caution in patient use.

Accuracy, Completeness, and Clarity of an AI-Based Chatbot for the EAU Neuro-Urology Guidelines.

This study aimed to externally validate the performance of the European Association of Urology (EAU) Guidelines Bot in neuro-urology by assessing the accuracy, completeness, and clarity of chatbot-generated answers to guideline-based questions and to compare its performance with that of a general-purpose large language model (ChatGPT 5.5). A cross-sectional validation study was conducted using 47 questions derived from the EAU Neuro-Urology Guidelines. Each question was linked to a specific recommendation and classified by recommendation strength (strong vs weak). Questions were independently submitted to both the EAU Guidelines Bot and ChatGPT 5.5 without additional prompting. Two expert urologists independently evaluated each response for accuracy, completeness, and clarity using a five-point Likert scale; discrepancies were resolved by a third reviewer. Overall, 45 questions (95.7%) were linked to strong recommendations and two (4.3%) to weak recommendations. The EAU Guidelines Bot and ChatGPT 5.5 achieved identical mean accuracy scores (4.96 ± 0.20), with all responses rated as highly accurate (Likert 4-5). ChatGPT 5.5 indicated significantly higher completeness scores than did the EAU Guidelines Bot (4.74 ± 0.44 vs 4.57 ± 0.54; p = 0.011), whereas clarity scores were not significantly different (4.83 ± 0.38 vs 4.77 ± 0.43; p = 0.083). High-quality completeness was observed in 46/47 EAU Guidelines Bot responses (97.9%) and 47/47 ChatGPT responses (100%). Score discrepancies between systems were identified in ten of 47 questions (21.3%) and were limited to completeness and clarity domains. Performance remained uniformly high across recommendation grades, with no meaningful differences observed. The EAU Guidelines Bot showed excellent accuracy, completeness, and clarity when applied to neuro-urology guideline-based questions. Its performance was comparable to that of ChatGPT 5.5, with both systems providing highly accurate guideline-concordant responses. Although ChatGPT 5.5 generated more comprehensive answers, the EAU Guidelines Bot maintained closer adherence to the original guideline recommendations. Although not a substitute for clinical judgment, the tool appears to be a reliable adjunct for rapid access to evidence-based neuro-urological guidance.