Gemini-2.5-Flash provided the most reliable responses to common patient questions about rTKA generated by leading LLMs, highlighting the need for supervised integration of LLMs in patient education.
Abstract
Background
Robotic-assisted total knee arthroplasty (rTKA) is increasingly used because of its surgical precision. However, inconsistent outcomes and high costs often lead patients to seek additional information from artificial intelligence (AI) tools. Large language models (LLMs) such as ChatGPT-4o, Gemini-2.5-Flash, and DeepSeek-V3 are commonly used, but their reliability and readability in orthopaedics remain unclear.
Objectives
To compare the reliability, usefulness, quality, and readability of responses to common patient questions about rTKA generated by leading LLMs.
Methods
Three LLMs answered 20 frequently asked patient questions (n = 20) identified through Google Trends and expert validation. Three orthopaedic specialists (n = 3) evaluated reliability, usefulness, and overall quality using validated scales, while readability was assessed with standard indices.
Results
Inter-rater reliability was good to excellent (ICC = 0.728-0.879). Gemini-2.5-Flash achieved significantly higher reliability and usefulness scores than ChatGPT-4o and DeepSeek-V3 (all p < 0.05). ChatGPT-4o and DeepSeek-V3 produced more readable but less accurate content, revealing an inverse relationship between reliability and readability.
Conclusions
Gemini-2.5-Flash provided the most reliable responses, highlighting the need for supervised integration of LLMs in patient education.
Although Gemini 3 and DeepSeek demonstrated greater readability in selected comparisons, median responses across all models remained above recommended patient-education reading levels, and LLM-generated responses should be regarded as supplementary rather than standalone sources of patient information regarding RA-TKA.
U. Kolaç, Mazlum Veysel Sili, Orhan Mete Karademir et al.· Knee (Oxford)· 0 citations
There may be significant differences in how effectively LLMs support patients with surgical queries, particularly in areas needing detailed explanation, and usually required minimal clarification in areas needing detailed explanation.
T. Davis, B. Guevel, K. Logishetty et al.· Annals of the Royal College...· 0 citations
Introduction Large language models (LLMs) are increasingly used to answer patient questions, but the readability and information quality of patellar dislocation guidance remain unclear. Methods A series of clinical questions covering etiopathogenesis, mechanisms, clinical manifestations, diagnosis, treatment, complicat...
Shuo Liu, Xi-Yang Sun, Yun-Fei Ma et al.· Frontiers in Digital Health· 0 citations
AI chatbots can generate information with potential clinical relevance in anterior implant dentistry; however, variability in informational reliability persists and expert supervision remains essential before integrating such tools into clinical education.
Dalndushe Abdulai, Raghıb Suradı, Mehran Moghbel· Journal of Health Sciences a...· 0 citations
LLMs performed well in several aspects of patient communication and education with support from high clinician ratings, accuracy, and readability scores, however, evidence for documentation and decision support remained limited.
Ivan A. Garces, Andres G. Wong, J. Brutti et al.· JB & JS open access· 0 citations