Large Language Models vs Expert Response: A Comparative Analysis of Patient Education Quality After Skull Base Surgery

Abstract

Objective: Patients undergoing endoscopic skull base surgery (ESBS) often seek postoperative guidance. Large language models (LLMs) are increasingly accessible resources, but their quality in this setting remains poorly studied. This study compared LLM-generated responses with expert responses for common postoperative ESBS questions. Methods: Eight common postoperative questions were answered by seven LLMs (ChatGPT-o1, Gemini, Claude, Meta, GROK, DeepSeek, Copilot) and expert surgeons. Eight blinded raters evaluated responses for understandability and actionability using the Patient Education Materials Assessment Tool (PEMAT), while accuracy was graded on a 5-point Likert scale. Readability was assessed using Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease Score (FRES). ANOVA with Tukey HSD tested group differences. Results: Most LLM-generated responses showed accuracy comparable to expert responses (p>0.05), except ChatGPT-o1, which scored lower (p=0.006). Significant differences were found in understandability (F(7,56)=23.42, p<0.001, η²=0.75) and actionability (F(7,56)=21.06, p<0.001, η²=0.72). Gemini, GROK, Meta, and DeepSeek achieved the highest PEMAT scores. Expert responses scored lowest on PEMAT. Expert and Claude required college-level reading, whereas ChatGPT-o1 and Copilot produced grade-school–level content. Gemini, GROK, Meta, and DeepSeek produced high-school–level text. Conclusion: LLMs generated responses that were more understandable, actionable, and readable than expert responses while maintaining comparable accuracy. LLMs may improve postoperative counseling after skull base surgery, though expert oversight remains essential.