Evaluation of Artificial Intelligence Chatbots for Preoperative Counseling in Patients with Neurological Disorders
Abstract
Objective: The quality of information provided to patients with neurological disorders using large language model (LLM)-based artificial intelligence (AI) chatbots has not been sufficiently investigated. The aim of this study was to evaluate and compare the quality of responses generated by ChatGPT, Google Gemini, and Microsoft Copilot to frequently asked preoperative questions from patients with neurological disorders. Methods: Twenty frequently asked preoperative questions were identified across four neurological disorders: multiple sclerosis (MS), epilepsy, Parkinson’s disease, and dementia. Each question was submitted to ChatGPT, Gemini, and Copilot using identical prompts in newly initiated sessions. Responses were evaluated in a blinded manner by a panel of eight neurologists and two anesthesiologists. Clinical accuracy, safety, and adequacy were assessed using a 5-point Likert scale. Inter-rater agreement was assessed using the intraclass correlation coefficient (ICC). Results: ChatGPT achieved the highest overall scores across all disease groups. Significant differences were observed among chatbots in the overall analysis ( χ ²(2)=40.635, P<0.001, Kendall’s W=0.423). Post-hoc analyses demonstrated that ChatGPT significantly outperformed both Gemini (r=0.705, p<0.001) and Copilot (r=0.747, p<0.001), whereas no significant difference was found between Gemini and Copilot (p=0.876). The lowest scores for all chatbots were observed in the MS group. Overall inter-rater agreement was moderate (ICC=0.571; 95% CI, 0.411–0.712). No response was classified as potentially harmful or misleading. Conclusion: All three AI chatbots provided generally high-quality responses to questions from patients with neurological disorders. ChatGPT demonstrated superior performance compared with Gemini and Copilot. Although these tools may support patient education, they should complement rather than replace physician-guided counseling.