EP4.9 Comparing Artificial Intelligence Performance to Hip Preservation Experts on Consensus-Based Hip Preservation Questionnaire
TL;DR
Evaluating the performance of three AI platforms versus a panel of international hip preservation experts in answering guideline-based questions related to femoroacetabular impingement syndrome, hip dysplasia, and microinstability of the hip shows that the knowledge of AI is rapidly approaching and/or exceeding that of medical subspecialists.
Abstract
Abstract Purpose Artificial intelligence (AI) is increasingly utilized in medical education and clinical contexts, yet few studies compare the newest large language models (LLMs) to subspecialized experts on guideline-based medical knowledge. The purpose of this study was to evaluate the performance of three AI platforms versus a panel of international hip preservation experts in answering guideline-based questions related to femoroacetabular impingement syndrome, hip dysplasia, and microinstability of the hip. Methods A 21-item multiple-choice questionnaire was developed, each containing a single correct answer based on published consensus guidelines. The survey was distributed to a panel of hip preservation specialists, and 10 expert responses were included for analysis. Three LLMs (ChatGPT 5.2, Gemini 3, and Claude 4.5 Sonnet) were prompted using the same questionnaire, with each LLM performing 3 runs per item. Outcomes included overall accuracy, percent agreement, Fleiss’ Kappa, generalized linear mixed-effects modeling, and qualitative assessment of AI answer justifications. Results Expert accuracy was 90.5% (95% CI 87.1 – 93.9), compared to 100% (p = 0.004) for Gemini, 98.4% (p = 0.016) for ChatGPT, and 96.8% (p = 0.053) for Claude. Expert percent agreement was 42.9% and Fleiss’ Kappa was 0.769; alternatively, AI intra-item percent agreement was 100% (Gemini) and 95.2% (ChatGPT and Claude), and Fleiss’ Kappa could not be accurately calculated due to high consistency. ChatGPT and Claude provided thorough justifications for even incorrect responses, and Gemini demonstrated consistent formatting deviations despite 100% accuracy. Conclusion Although human performance was good, all 3 LLMs demonstrated high accuracy and consistency when answering the questionnaire and outperformed the expert panel. In structured, verifiable question sets, it is evident that the knowledge of AI is rapidly approaching and/or exceeding that of medical subspecialists. LLMs are likely to serve as an effective adjunct in medical education, and limitations to AI’s implementation into clinical practice should be explored.