Skip to content
Review Open access

Large Language Model Benchmarks in Medical Tasks

Oct 2024 · Medicine Advances · Vol 4, pp. 316 - 341 · 34 citations · ⚡ 1 influential · 366 references
Computer Science

TL;DR

A comprehensive survey of various benchmark datasets used in medical LLM tasks, focusing on various aspects of medical knowledge such as electronic health records, doctor–patient dialogues, medical question answering, and medical image captioning, is presented.

Abstract

With the increasing application of large language models (LLMs) in the medical domain, evaluating these models' performance using benchmark datasets has become crucial. This paper presents a comprehensive survey of various benchmark datasets used in medical LLM tasks. These datasets span multiple modalities including text, image, and multimodal benchmarks, focusing on various aspects of medical knowledge such as electronic health records, doctor–patient dialogues, medical question answering, and medical image captioning. The survey categorizes the datasets by modality and examines their significance, data structure, and roles in model development and evaluation across tasks such as diagnostic support, report generation, and predictive decision support. Representative resources include Medical Information Mart for Intensive Care III (MIMIC‐III), MIMIC‐IV, BioASQ, PubMedQA, and CheXpert, which provide data and evaluation settings for research in clinical NLP, medical question answering, and chest‐radiograph interpretation. This paper summarizes the challenges and opportunities in leveraging these benchmarks for advancing multimodal medical intelligence, emphasizing the need for datasets with a greater degree of language diversity, structured omics data, and innovative approaches to synthesis. This synthesis is intended to inform future research on the applications of LLMs in medicine and medical artificial intelligence.

Read PDF

Similar papers

Review

The Application of Large Language Models in Medical Question‑Answering Robots

This review explores the transformative impact of Large Language Models on medical question-answering robots and addresses critical challenges faced by LLM-driven systems, such as limited interpretability, restricted applicability across medical domains, and serious concerns about patient privacy.

Zhao-Xu Wang · 0 citations
#artificial intelligence Review Sep 2026

A primer on evaluation methods for large language models in healthcare

This article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs by describing underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question.

S. E. McKinney, Phuc Vu, Samuel A. Justice et al. · 0 citations
#artificial intelligence Preprint Sep 2026

MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain

MMTClinic is presented, a benchmark designed to evaluate large language models (LLMs) on complex reasoning and question-answering tasks involving clinical time-series and reveals notable differences in model performance across tasks, languages, and modalities, highlighting current limitations in clinical reasoning capa...

Sourav Malakar, Harshit Nigam, Akash Ghosh et al. · 0 citations
Preprint Aug 2026

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

This work introduces MedReaMM, a benchmark specifically designed to evaluate models'ability to synthesize heterogeneous clinical evidence consisting of detailed patient histories alongside multiple medical images into accurate differential diagnoses under a complete-information paradigm.

Lai Wei, Yu-Chao Chen, Zhenbiao Cao et al. · 0 citations
Open access Oct 2026

Entity-centric evaluation of large language model responses for medical question-answering tasks

EntQA is proposed, a reference-free, entity-centric metric that evaluates how well LLM-generated responses retain clinically relevant biomedical concepts from patient backgrounds and diagnostic questions that outperforms conventional overlap- and embedding-based metrics that frequently exhibited weak or negative correl...

Yi Liu, V. B. Kolachalama · 0 citations
Review Open access Oct 2026

The impact of a domain-specific harness on large language model evaluations of medical necessity in oncology.

740 Background: While large language models (LLMs) have been effective at medical domain question and answer, their effectiveness and sufficiency in determining the medical necessity (MN) of complex therapeutics in oncology for a given patient’s clinical context is unclear. Recent advances in artificial intelli...

B. Chaudhry, Andrew T. Yue, Lisa Tran · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.