Augmenting the comprehensiveness and quality of serious illness documentation using large language models.

Abstract

393 Background: Inpatient clinicians need timely access to prior serious illness conversations (SICs) for patients with advanced cancer. Although structured electronic health record (EHR) modules exist to document SICs, they are infrequently completed, and most SIC information remains embedded in free-text clinical notes. We evaluated whether large language models (LLMs) could synthesize longitudinal clinical notes into comprehensive, clinically relevant SIC summaries. Methods: The BRIDGE-SIC pilot clinical trial enrolled 58 patients with solid tumors who were admitted to an academic hospital with a predicted 90-day mortality ≥40%. A HIPAA-compliant LLM identified SIC documentation from the prior 6 months and summarized content across 5 domains (illness understanding, prognosis, hopes, worries, and social supports). In this post-hoc analysis, Epic SIC modules and LLM-generated summaries were compared for comprehensiveness, accuracy, recency, and longitudinality. Results: Of 936 LLM summary points, 901 (96%) accurately reflected underlying documentation. Inaccuracies primarily stemmed from misattributing the clinician perspective as patient opinion, overinterpreting patient opinions, or citing incorrect source notes. Inaccuracies were not clinically significant, and there was no instance of hallucinated clinical data. LLM summaries captured SIC content across 4 of 5 domains for all patients (100%, 58/58) and all 5 domains for 88% (51/58). Fourteen percent (8/58) had SICs documented within the Epic module; most (63%; 5/8) were documented by a palliative care clinician. In contrast, LLM summaries synthesized SIC content from a mean of 10 notes per patient (SD=3.2) spanning a mean of 139 days (SD=144), of which 10% (SD=13%) were authored by palliative care clinicians. Compared with Epic SIC modules, LLM summaries incorporated SIC information documented a mean of 67 days more recently (SD=129). Conclusions: LLMs can accurately synthesize longitudinal free-text notes into SIC summaries that are more comprehensive and timely than the Epic SIC module while including data from a more diverse set of providers. Clinical trial information: NCT07147023 . A comparison between an Epic SIC module and an LLM-generated SIC summary for “hopes.” SIC module [date] Be comfortable; Be mentally aware LLM summary [date] Hopes for tumor shrinkage, symptom palliation, and increased life expectancy. Clear about not wanting heroic measures at end of life, with DNR/DNI code status.