Evaluation of a Large Language Model Discharge Summary Hospital Course Tool: Improved Quality but Longer Documentation Time
Abstract
Abstract Background No studies have examined the effect of artificial intelligence (AI)-generated hospital courses on discharge summary documentation time. Objective This study aimed to evaluate the effect of an AI-generated hospital course tool on discharge summary edit time, quality, and clinician perceptions. Methods Observational study between a “pre-AI period” (March 16, 2025–November 19, 2025) and a “post-AI period” (November 20, 2025–February 11, 2026). An AI hospital course tool using a GPT-4.1 model to generate a draft hospital course integrated into mandatory discharge summary templates on November 20, 2025. We included 8298 hospitalized adults cared for by hospital medicine clinicians. The primary outcome was discharge summary edit time, defined as time from note creation to signature across all sessions. A subset of 27 encounters was reviewed for cohesiveness, comprehensiveness, conciseness, potential for harm, and overall quality. Thirty-six clinicians completed a survey assessing usability, efficiency, trustworthiness, and quality. Results Implementation of the AI tool was not associated with a significant change in discharge summary edit time between study periods (6.60 [IQR: 3.4–13.5] vs. 6.28 [IQR: 3.2–13.0] min, p = 0.11). Among 2206 encounters in the post-AI period, edit time was longer when the AI tool was used (9.20 [IQR: 4.9–16.3] vs. 4.93 [IQR: 2.5–9.7] min, p < 0.001). In multivariable analysis, the AI tool was associated with a 32% increase in edit time (95% CI: 24–41, p < 0.001; adjusted R 2 = 0.34). Faculty review found AI-generated summaries had higher quality and lower harm scores but were less concise. Thirty-one (86.1%) clinicians agreed the tool improved efficiency. Conclusion Discharge summary edit time did not differ between study periods but was longer when the AI tool was used. However, there was increased perception of efficiency. The tool improved summary quality and reduced potential harm but may increase documentation time.