Large language models for analyzing contrast-enhanced ultrasound reports of pancreatic cystic lesions
TL;DR
LLMs show promising capability in interpreting CEUS reports of PCLs, with no statistically significant difference from senior radiologists in this dataset, and their integration significantly improves junior clinicians’ interpretation of CEUS reports.
Abstract
Background Pancreatic cystic lesions (PCLs) require precise imaging characterization to guide clinical management. Contrast-enhanced ultrasound (CEUS) reports contain operator-dependent narratives that challenge clinicians. Large language models (LLMs) show potential in medical text analysis but lack validation for pancreatic CEUS interpretation. This study primarily aimed to evaluate the diagnostic accuracy of LLMs in interpreting Chinese CEUS reports of PCLs. Methods We retrospectively analyzed 80 pathologically confirmed PCLs (21 benign, 25 borderline malignant, 34 malignant). Four LLMs (GPT-4o, Claude 3.7 Sonnet, Gemini 2.0, DeepSeek-R1) and eight radiologists (senior/junior =4:4) independently interpreted reports under three input modalities: grayscale-only (IM1), grayscale + CEUS (IM2), and demographics + grayscale + CEUS (IM3). A weighted scoring system (0–100 points per case, yielding a maximum total of 8,000 points) quantified alignment with pathology-defined categories (benign/borderline/malignant). LLM errors were categorized into four reasons. Junior radiologists reinterpreted cases with LLM assistance. Results Under CEUS input conditions (IM2/IM3), LLMs showed no statistically significant difference from senior radiologists and significantly outperformed junior radiologists. The median score per case (out of 100), after averaging across the four LLMs, increased with input complexity (IM1: 47.50; IM2: 53.75; IM3: 73.75). Diagnostic accuracy varied by pathology: malignant lesions scored highest, while benign serous cystic neoplasms scored lowest due to suboptimal CEUS visualization and LLMs’ knowledge gaps. LLM guidance elevated junior radiologists’ accuracy to senior levels. Conclusions LLMs show promising capability in interpreting CEUS reports of PCLs, with no statistically significant difference from senior radiologists in this dataset. Their integration significantly improves junior clinicians’ interpretation of CEUS reports. These findings support further investigation of LLMs as potential auxiliary tools for ultrasound text analysis, though external validation is needed.