How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content

Abstract

Patients recovering from lumbar fusion increasingly seek guidance from large language model (LLM) chatbots when their surgeon is unavailable, but the accuracy, completeness, and safety-related content of such responses have not been compared across models for the postoperative period. To compare the accuracy, quality, safety-related content, and readability of responses from ChatGPT, Gemini, and Claude to frequently asked postoperative lumbar fusion questions, and to identify the question types for which responses are weakest. Thirty questions, compiled by three neurosurgeons and grouped into six clinical categories, were submitted to each model, each with a standardized, patient-oriented prompt requesting warning signs. Ninety anonymized, randomized responses were rated independently by three blinded neurosurgeons for accuracy and four quality subdomains (clarity, completeness, relevance, and patient safety) on 4-point scales. Readability (Flesch–Kincaid, Flesch Reading Ease, and Gunning Fog), word count, and responses meeting an operational screening criterion for potential harm were also assessed. Models were compared with Friedman and Holm-corrected pairwise tests. Accuracy was high for all models without significant differences. Quality, clarity, completeness, relevance, and patient safety differed globally, but effect sizes were small (Kendall's W : 0.10–0.21) and, after correction across the six outcome families, only quality and completeness remained significant, with Gemini scoring lower than ChatGPT and Claude. Gemini had the most favorable readability. ChatGPT responses were the shortest (mean 294 vs. 462 and 445 words); length was unrelated to completeness or readability. Two Gemini responses (6.7%) and none from ChatGPT or Claude met a post hoc operational screening criterion for potential harm (a score of ≤2 on accuracy or patient safety from at least one of three raters, without concurrence from the other two); no response was adjudicated as actually harmful. In exploratory question- and category-level analyses, responses tended to be rated highest for standardized questions and lowest for those requiring individualized judgment or interpretation of wound-related symptoms. Responses from contemporary LLMs to common postoperative lumbar fusion questions were generally rated as accurate, with small between-model differences and a model-level divergence between readability and completeness. Because safety-related content was explicitly prompted and no patient outcomes were assessed, these findings describe the quality of safety-related information rather than clinical safety. LLMs may support postoperative patient education but should not replace physician guidance, particularly for warning signs and patient-specific recovery decisions.