Evaluation of the performance of ChatGPT-4o on oral surgery-related questions in the Japanese National Dental Examination.
TL;DR
The presence of multiple visual materials significantly reduced accuracy, with panoramic radiographs and dental models showing the strongest negative effects and the development of multimodal models capable of integrating both text and images.
Abstract
Artificial intelligence (AI) has advanced rapidly in healthcare, with large language models (LLMs) like ChatGPT-4o showing potential in education and clinical support. This study evaluated the performance of ChatGPT-4o on oral surgery-related questions from the Japanese National Dental Examination, focusing on how visual materials impact accuracy. Questions were categorized by type (general knowledge vs clinical practice), number of correct answers required, and presence of visual materials (e.g., radiographs, models). The comparison of correct and incorrect answers according to the number of visual materials was performed using the Mann-Whitney U-test, while logistic regression was used to assess the influence of specific visual content types. ChatGPT-4o performed well on general knowledge questions but showed lower accuracy on questions requiring clinical decision-making . The presence of multiple visual materials significantly reduced accuracy, with panoramic radiographs and dental models showing the strongest negative effects (odds ratios 0.46 and 0.45, respectively; both P = 0.002). Other visual materials, such as computed tomography/magnetic resonance imaging scans, had no significant impact. These findings highlight the strengths of ChatGPT-4o in processing structured textual data and its limitations in interpreting visual or procedural content. Enhancing AI-assisted dental education will require domain-specific training data and the development of multimodal models capable of integrating both text and images. This study informs future applications of AI in oral surgery education.