Artificial intelligence in cardiology: evaluating the accuracy of ChatGPT-o1-preview in a medical specialisation exam

Abstract

Recent advancements in artificial intelligence (AI) language models, such as ChatGPT, have sparked interest in their application across various domains, including medicine. This study evaluates the performance of the ChatGPT-o1-preview model on the Polish Specialist Examination (PES) in cardiology. A total of 120 single-choice questions from the Spring 2023 PES cardiology exam were analysed. Each question was submitted to the model five times in separate sessions using a standardised prompt. We assessed the accuracy of the responses and the model’s consistency by introducing a “Confidence Index of Language Model” (defined as the ratio of the frequency of the most commonly selected answer to the total number of sessions). Questions were classified according to Bloom’s Taxonomy, and statistical relationships between model performance and question characteristics were analysed. ChatGPT-o1-preview provided 97 correct answers (80.83%), well above the passing threshold of 60%. Statistically significant correlations were observed between answer accuracy and difficulty indices (both the Human Difficulty Index and the Confidence Index of Language Model). Questions that were more difficult for human examinees also proved more challenging for the model. No significant differences in model performance were found based on question type (clinical versus other) or content (comprehension and critical thinking versus memory). ChatGPT-o1-preview successfully completed a specialised cardiology exam. It is essential to conduct further, more extensive research based on a significantly larger and more diverse set of questions. Only then will it be possible to reliably determine the true potential and limitations of AI in medical applications.