Accuracy and Error Patterns of Generative AI on the Korean National Dental Hygienist Licensing Examination Across Prompt Engineering Strategies

Abstract

Objective: This study aimed to compare GPT-4o accuracy across five prompt engineering strategies using items from the Korean National Dental Hygienist Licensing Examination and to examine differences by question domain and type. Methods: A total of 399 items from the 2024-2025 examinations were classified by domain (Basic/Clinical) and type (Recall/Reasoning) and presented to GPT-4o (GPT-4o-2024-11-20, temperature=0) under five prompt conditions. Differences in accuracy across prompt conditions were assessed using Cochran’s Q test. For reasoning-type items, exploratory pairwise comparisons were performed using exact McNemar tests with Holm correction. Results: Overall accuracy ranged from 75.2% to 78.9%, exceeding the overall threshold of 60% under all prompt conditions. Cochran’s Q showed no significant overall difference among the five prompt conditions (Q=6.462, p=0.167). For reasoning-type items (n=80), the combined prompt yielded the highest accuracy(75.0%) compared with 65.0%-67.5% for the other conditions. However, the overall difference among prompt conditions was not statistically significant (Q=6.759, p=0.149), and no pairwise comparison remained significant after Holm correction. Qualitative analysis identified recurring hallucinations involving Korea-specific regulations. Conclusions: GPT-4o achieved accuracy exceeding the 60% overall threshold regardless of prompt strategy, suggesting that the overall effect of prompt engineering was limited. Although the combined prompt showed a favorable trend for reasoning-type items, statistical evidence for this effect was limited. These findings highlight the importance of AI literacy for the critical evaluation of AI-generated rationales.