Performance and concordance of contemporary large language models on Royal Australasian College of Physicians examination questions: an exploratory analysis.

Abstract

We assessed six publicly available large language models (LLM) on Royal Australasian College of Physicians (RACP) adult medicine written examination questions. In a 30-question set with answers, models achieved 93% correct responses overall, and accuracy was higher for text-based than visual-media questions. In a 170-question exploratory set without an answer key, complete inter-model agreement occurred in 86% of questions but was lower for visual-media than text-based questions and for scenario-based than knowledge-based questions. Contemporary LLM performed strongly on RACP-style written questions, but interpretation of visual media and complex clinical vignettes remains a limitation.