The daily dose: Early usability of an LLM tool for patient summaries and trial matching in radiation oncology

Abstract

Background Radiation oncology workflows generate large volumes of electronic health record (EHR) data requiring daily synthesis. Large language model (LLM)-based automation is promising, but workflow-embedded implementations at scale remain limited. We describe the design and early usability and adoption evaluation of The Daily Dose (TDD), an LLM-driven system for automated clinical summarization and trial identification in radiation oncology. Materials and methods TDD delivers physician-specific email summaries each morning across three Mayo Clinic campuses using RadOnc-GPT (GPT-4o) to generate EHR-derived patient summaries and identify potentially eligible clinical trials for new or consult visits. One month post-deployment, an anonymous cross-sectional survey adapted from the System Usability Scale and Technology Acceptance Model was administered to all recipients. Results Fifty-five of 110 users responded (50%); 94.5% were in radiation oncology and 69.1% were attending physicians. Overall, 83.6% used TDD at least several times per week. Mean domain scores (5-point Likert) were 3.89 ± 1.04 for usability and satisfaction, 3.43 ± 1.24 for perceived usefulness, and 3.80 ± 1.17 for impact and future use. Satisfaction was significantly associated with perceived time savings (p < 0.001); 27% estimated saving ≥10 min daily. Internal consistency was high (α = 0.97). Free-text responses highlighted improved preparedness and patient-context awareness but noted occasional inaccuracies and imperfect trial matching. Conclusion In this early usability and adoption evaluation, a workflow-integrated LLM summarization tool was widely adopted and generally favorably perceived. These findings reflect user perceptions; objective validation of summary accuracy, trial-matching performance, and workflow efficiency is needed to establish clinical impact.