Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how clinicians retrieve and synthesize information from patient records. Their safety and utility depend on rigorous evaluation, yet existing benchmarks are manually curated, costly to update, and rapidly become obsolete with evolving technological advancements. We present a scalable framework that automatically generates question--answer pairs from longitudinal EHR notes. Nineteen clinicians validate the benchmark generator, producing the Benchmark for Retrieving Information in EHRs (BRIE), a continuously maintainable evaluation dataset. Across nine LLMs and five inference strategies, state-of-the-art systems frequently omit clinically important information, particularly for questions requiring synthesis across multiple documents and encounters. Because the generator itself is validated, BRIE supports evaluations that static benchmarks cannot, including the generation of multiple answers that reflect variation in clinician reasoning for robust performance assessment and continuously refreshing benchmark content to guard against leakage. Our results demonstrate that scalable benchmark generation enables rigorous, up-to-date evaluation of clinical LLMs as they are deployed in rapidly evolving healthcare settings.
Health care organizations increasingly require structured, patient-level clinical variables for treatment decisions, operational workflows, quality measurement, and clinical trial screening. Relevant information is often fragmented across heterogeneous electronic health record (EHR) systems, unstructured formats, a...
Timothy J. Stuhlmiller, AJ Rabe, Jeff Rapp et al.· AI in Precision Oncology· 0 citations
In hospital workflows, electronic health records (EHRs) are often noisy, and may not contain the evidence needed to confirm events or measurements referenced in a clinical query. Even when database retrieval succeeds, clinical agents can overlook such discrepancies and return plausible but unsupported answers. We intro...
Yi-Tong Qiao, Yan-Cheng Jin, Lei Liu et al.· arXiv.org· 0 citations
Large language model (LLM) assistants are being deployed to clinicians across health systems, and judgments about their readiness rest largely on benchmark scores, most of them derived from examination questions or curated cases. A benchmark predicts performance in deployment only to the extent that its items resemble...
K. Vishwanath, Hai-Tong Lin, A. Alyakin et al.· 0 citations
BACKGROUND
Precision medicine requires delivery and tracking of genome-informed risk assessments (GIRAs), often documented in unstructured electronic health record (EHR) notes, making large-scale evaluation reliant on labor-intensive manual chart review. Large language models (LLMs) offer a promising approach to automa...
Yi Xin, K. W. Davis, Wu-Chen Su et al.· JAMIA Journal of the America...· 0 citations
A comprehensive framework for transforming LLMs into trustworthy medical specialists is presented, and the evolution toward agentic AI systems that orchestrate them for autonomous, collaborative clinical decision-making is explored.
Kiduk Kim, Jeong Min Song, Dong Yeong Kim et al.· Cell Reports Medicine· 0 citations
Empirical evidence is provided on how retrieval, embedding, and prompting choices influence clinical information extraction performance and practical guidance is offered for researchers and healthcare informatics practitioners seeking to optimise retrieval-augmented large language model systems for extracting clinicall...
Dinithi S. Vithanage, Quang Vinh Duong, Chao Deng et al.· Journal of Healthcare Inform...· 0 citations