Comparison of chatbots for patient information on head and neck imaging techniques in terms of scientific quality and readability

Abstract

Aims: This study aimed to evaluate the scientific quality and readability of the responses of ChatGPT and Microsoft Copilot to patient questions about head and neck imaging methods in dentistry. Methods: Twenty patient-focused questions were directed to both chatbots. Responses were evaluated independently by two experts using a 7-point Likert scale, and readability levels were analyzed using the Flesch Reading Ease Score and Flesch-Kincaid Grade Level (FKGL) indices. Inter-expert agreement was assessed using Cohen’s Kappa, and differences between chatbots were assessed using an independent-samples t-test. Results: Statistically significant but weak agreement was observed among experts (κ=0.399, p<0.001). No significant difference was found between ChatGPT and Microsoft Copilot in terms of Flesch Reading Ease Score index and expert scores (p>0.05). However, FKGL scores were significantly higher in Microsoft Copilot (p=0.025), indicating more linguistically complex responses. Conclusion: ChatGPT and Microsoft Copilot demonstrated similar scientific quality and readability based on FRES scores. However, Microsoft Copilot generated more linguistically complex responses than ChatGPT, as reflected by its significantly higher FKGL scores, indicating that a higher educational level may be required for comprehension. In conclusion, AI-based chatbots can be useful as supportive tools in patient information and education, but expert supervision remains crucial in this area.