Oct 2024· Medicine Advances· Vol 4, pp. 316 - 341· 34 citations· ⚡ 1 influential· 366 references
Computer Science
TL;DR
A comprehensive survey of various benchmark datasets used in medical LLM tasks, focusing on various aspects of medical knowledge such as electronic health records, doctor–patient dialogues, medical question answering, and medical image captioning, is presented.
Abstract
With the increasing application of large language models (LLMs) in the medical domain, evaluating these models' performance using benchmark datasets has become crucial. This paper presents a comprehensive survey of various benchmark datasets used in medical LLM tasks. These datasets span multiple modalities including text, image, and multimodal benchmarks, focusing on various aspects of medical knowledge such as electronic health records, doctor–patient dialogues, medical question answering, and medical image captioning. The survey categorizes the datasets by modality and examines their significance, data structure, and roles in model development and evaluation across tasks such as diagnostic support, report generation, and predictive decision support. Representative resources include Medical Information Mart for Intensive Care III (MIMIC‐III), MIMIC‐IV, BioASQ, PubMedQA, and CheXpert, which provide data and evaluation settings for research in clinical NLP, medical question answering, and chest‐radiograph interpretation. This paper summarizes the challenges and opportunities in leveraging these benchmarks for advancing multimodal medical intelligence, emphasizing the need for datasets with a greater degree of language diversity, structured omics data, and innovative approaches to synthesis. This synthesis is intended to inform future research on the applications of LLMs in medicine and medical artificial intelligence.
This review explores the transformative impact of Large Language Models on medical question-answering robots and addresses critical challenges faced by LLM-driven systems, such as limited interpretability, restricted applicability across medical domains, and serious concerns about patient privacy.
This article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs by describing underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question.
S. E. McKinney, Phuc Vu, Samuel A. Justice et al.· arXiv.org· 0 citations
MMTClinic is presented, a benchmark designed to evaluate large language models (LLMs) on complex reasoning and question-answering tasks involving clinical time-series and reveals notable differences in model performance across tasks, languages, and modalities, highlighting current limitations in clinical reasoning capa...
Sourav Malakar, Harshit Nigam, Akash Ghosh et al.· 0 citations
This work introduces MedReaMM, a benchmark specifically designed to evaluate models'ability to synthesize heterogeneous clinical evidence consisting of detailed patient histories alongside multiple medical images into accurate differential diagnoses under a complete-information paradigm.
Lai Wei, Yu-Chao Chen, Zhenbiao Cao et al.· 0 citations
EntQA is proposed, a reference-free, entity-centric metric that evaluates how well LLM-generated responses retain clinically relevant biomedical concepts from patient backgrounds and diagnostic questions that outperforms conventional overlap- and embedding-based metrics that frequently exhibited weak or negative correl...
Yi Liu, V. B. Kolachalama· PLOS Digital Health· 0 citations
740
Background:
While large language models (LLMs) have been effective at medical domain question and answer, their effectiveness and sufficiency in determining the medical necessity (MN) of complex therapeutics in oncology for a given patient’s clinical context is unclear. Recent advances in artificial intelli...
B. Chaudhry, Andrew T. Yue, Lisa Tran· JCO Oncology Practice· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.