Review Open access

Evaluating Large Language Models as Educational Tools: A Multi-Model Analysis of Performance on the Turkish Medical Specialty Examination (TUS) Ophthalmology Questions Özet:

Burhan Başkan Yusuf Evcimen Özgür Bülent Timuçin İffet Yarımağa Celal Sağmış

Jul 2026 · Muş Alparslan Üniversitesi Sağlık Bilimleri Dergisi · Vol 6, pp. 51-65 · 0 citations · 19 references

Abstract

This study aimed to comprehensively evaluate the performance of seven leading large language models (LLMs) from the 2024-2025 period on Turkish Medical Specialty Examination (TUS) ophthalmology questions, analyzing factors such as question type, chronology, and clinical area, while critically assessing the risk of data contamination and the technology's realistic potential in medical education. A set of 210 TUS ophthalmology questions (2015-2024) was presented to seven LLMs (OpenAI o1-preview, Claude 4.5 Sonnet, GPT-4o, Gemini 2.5 Pro, DeepSeek-R1, Llama 3 70B, Command R+) using a standardized protocol. Performance was statistically compared based on overall accuracy, question type (case-based vs. knowledge-based), chronological period (old: 2015-2019 vs. new: 2020-2024), and clinical subspecialty. All responses underwent blinded, expert qualitative coding for error type and confidence. OpenAI o1-preview achieved the highest overall accuracy (95.2%). Claude 4.5 Sonnet and GPT-4o demonstrated 100% accuracy on case-based questions. A significant performance gap existed between closed-source (average 91.0%) and open-source models (average 66.1%) (p

DOI Semantic Scholar

Read PDF

Similar papers

Open access Sep 2026

Evaluating the Accuracy of Large Language Models in Dentistry: A Multi-Model Study Using Clinical Questions From Turkey's Dental Specialty Exams.

While LLMs show strong potential in supporting dental education through standardised exams, their performance varies by model and question type, and further improvements are needed to enhance reliability across different dental disciplines.

N. Acar, Fatih Sengul, Periş Çelikel et al. · European journal of dental e... · 0 citations

Open access Aug 2026

Assessing the Educational Role of Large Language Models in Dental Training: A Decade-Long Analysis of Text-Based and Visual Questions in a National Examination.

LLMs demonstrate strong potential as supportive tools in dental education, particularly for text-based knowledge assessment, but their limited performance in visual question contexts highlights an important constraint for their integration into image-dependent domains such as dental diagnostics.

S. Taşyapan, Didem Özer · European journal of dental e... · 0 citations

Open access Aug 2026

Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo

Although the four LLMs generally provide accurate and pertinent information regarding type 2 diabetes, enduring limits in actionability and inconsistencies among models in content completeness and understandability restrict their effective use in diabetic patient education.

Zhaoxia Huang, Yuxin Bai, Jun-Yue Luo et al. · Frontiers in Public Health · 0 citations

Oct 2026

Benchmarking Large Language Models (LLMs) Against Medical Students Using Psychometrically Evaluated Authentic Clinically Oriented Anatomy Multiple-Choice Questions (MCQs) at the Apply and Analyze Levels of Bloom's Taxonomy.

Large language models (LLMs) are increasingly used by medical students as study aids, yet evidence regarding their performance on higher-order, clinically oriented anatomy questions relative to that of students-particularly at the Apply and Analyze levels of Bloom's Taxonomy-remains limited. This study aimed to compare...

Serdar Babacan · Clinical anatomy (New York,... · 0 citations

Open access Sep 2026

Do Large Language Models Use the Clinical Vignette? A Question Ablation Study on the Orthopaedic In-Training Examination

Clinical vignettes were the primary contributor to improving LLM performance on OITE whereas images had minimal effect and closed-source models achieved substantially higher accuracy than open-source models, and all models performed above chance when provided answer options alone.

F. Gafoor, M. Syed, M. Halai et al. · medRxiv · 0 citations

Review Open access Sep 2026

Benchmarking Large Language Model Performance in Generating and Assessing Radiology Objective Structured Clinical Examination

This study provides a benchmark of LLM performance for radiology OSCE-style content generation and evaluation during a specific snapshot of artificial intelligence development (August 2024).

Ankush Ankush, Samriddhi Burman, Sydney Smith et al. · Radiology Advances · 0 citations