A Multidisciplinary Evaluation of ChatGPT and DeepSeek's Effectiveness in Answering Questions on Antibiotic Prophylaxis
TL;DR
These tools show potential as information resources for antibiotic prophylaxis in both periodontology and cardiology, but there are still areas for improvement, bearing in mind the study’s limitations.
Abstract
In recent years, artificial intelligence technologies have become a source of information for patients seeking answers to questions related to medicine and dentistry. Antibiotic prophylaxis is administered prior to invasive dental and medical procedures to prevent systemic complications. Patients who turn to artificial intelligence tools for information now require accurate and consistent responses based on current guidelines. This study aims to evaluate the accuracy and consistency of answers provided by two artificial intelligence applications (ChatGPT 4.0 and DeepSeek) to patient-generated questions about the use of antibiotic prophylaxis. Thirty frequently asked questions regarding antibiotic prophylaxis were identified through a literature review and clinical experience. These questions were posed to both artificial intelligence platforms and the responses were recorded. The responses were then independently evaluated by specialists in periodontology and cardiology using the modified Global Quality Scale. Differences between the platforms were analyzed using the Wilcoxon Signed Ranks Test and inter-rater agreement was assessed using the Cohen's kappa (κ) coefficient. The mean scale score was 4.56±0.56 for ChatGPT 4.0 and 4.55± 0.62 for DeepSeek. A statistically significant difference was found between the two experts only for ChatGPT, with the cardiologist assigning higher scores than the periodontist. Moderate but significant inter-rater agreement was observed only for DeepSeek (κ = 0.547). These tools show potential as information resources for antibiotic prophylaxis in both periodontology and cardiology, but there are still areas for improvement, bearing in mind the study’s limitations. However, the observed inter-specialty differences highlight the need for multidisciplinary validation and refinement. More comprehensive studies involving contributions from different specialties are required to enhance their reliability and accuracy.