Benchmarking Large Language Models (LLMs) Against Medical Students Using Psychometrically Evaluated Authentic Clinically Oriented Anatomy Multiple-Choice Questions (MCQs) at the Apply and Analyze Levels of Bloom's Taxonomy.
TL;DR
Current-generation LLMs achieved uniform, ceiling-level accuracy on clinically oriented anatomy MCQs at both the Apply and Analyze cognitive levels, significantly exceeding student performance, and were unaffected by the item difficulty and discrimination properties that meaningfully differentiated student performance.
Abstract
Large language models (LLMs) are increasingly used by medical students as study aids, yet evidence regarding their performance on higher-order, clinically oriented anatomy questions relative to that of students-particularly at the Apply and Analyze levels of Bloom's Taxonomy-remains limited. This study aimed to compare the accuracy of five LLMs (ChatGPT-5, Gemini 2.5 Pro, Claude, Microsoft Copilot, and DeepSeek R1) with that of medical students on authentic, psychometrically evaluated anatomy multiple-choice questions (MCQs) at the Apply and Analyze cognitive levels, and to qualitatively assess LLM response quality using a structured rubric. In this retrospective, cross-sectional study, 20 clinically oriented anatomy MCQs (10 Apply-level and 10 Analyze-level) administered at Bursa Uludag University Faculty of Medicine during the 2025-2026 academic year were identified from a pool of 397 questions and analyzed together with anonymized, aggregated student examination records. Each MCQ was presented once to all five LLMs using a standardized prompt. LLM and student accuracy were compared using Fisher's exact test. Item difficulty and discrimination indices were compared between Apply- and Analyze-level questions using independent-samples t-tests. LLM response quality was additionally scored using a six-domain LLM Anatomy Performance Assessment Rubric by two human observers and, in a cross-evaluation design, by peer LLMs. All five LLMs achieved perfect accuracy (100%, 20/20), significantly outperforming students, whose overall accuracy was 67.95% (p < 0.001). Student accuracy declined from the Apply level (74.79% ± 14.87%) to the Analyze level (61.13% ± 9.30%; p = 0.02), while LLM accuracy remained at ceiling across both levels (p < 0.001 for all comparisons). Analyze-level items showed significantly greater discrimination than Apply-level items (0.446 ± 0.070 vs. 0.248 ± 0.055; p < 0.001), despite being answered correctly by all LLMs regardless of difficulty. Because LLM accuracy showed zero variance, inter-model statistical comparisons were not applicable. Qualitative rubric evaluation showed all LLMs scored maximally on 4 of 6 domains (correct answer selection, Bloom's taxonomy appropriateness, scientific consistency, and absence of hallucination); differentiation emerged only in Anatomical Accuracy and Anatomical Reasoning, where Gemini 2.5 Pro and Claude scored highest (10.0/10) and Microsoft Copilot scored lowest (8.0/10) among human observers. Peer LLM evaluators rated model output as good as or better than human observers did across all comparisons, never lower (Spearman's ρ = 0.825, p = 0.086). Current-generation LLMs achieved uniform, ceiling-level accuracy on clinically oriented anatomy MCQs at both the Apply and Analyze cognitive levels, significantly exceeding student performance, and were unaffected by the item difficulty and discrimination properties that meaningfully differentiated student performance. However, a leniency bias among peer LLM evaluators raises concerns about the reliability of LLM-based peer assessment. These findings support the potential of LLMs as supplementary anatomy education tools while underscoring the continued need for expert oversight.