Structured prompting improves detection of interpretation and section-consistency errors across seven models and consistently enhances LLM-based error detection, closing the gap between open-source and commercial models and offering a pathway for privacy-preserving, on-premises QA.
Abstract
Errors in radiology reports are a major patient-safety concern and are difficult to detect with manual quality assurance (QA). Large language models (LLMs) can assist, but generic prompting does not reflect radiologists’ structured, section-based workflows. To develop and evaluate RadCoT (Radiological Chain-of-Thought), a domain-specific prompting framework aligning LLM reasoning with radiological review workflows, and to assess whether it enables open-source models to approach commercial benchmarks for error detection. In this retrospective study, 1,170 clinician-validated errors were extracted from a departmental QA repository (January 2021–December 2024), corresponding to 900 error-containing reports. An additional 300 error-free reports served as controls, yielding 1,200 reports balanced across radiography, ultrasound, CT, and MRI. Errors were categorized into five types by experienced radiologists. Seven LLMs were evaluated using standard prompting and the six-step RadCoT framework. Micro-averaged precision, recall, and F1 were computed at the error-instance level. Error type, modality-specific performance, and inference time were analyzed. RadCoT significantly improved the mean micro-averaged F1 across all models from 0.77 ± 0.06 (standard) to 0.85 ± 0.06 (RadCoT; p = 0.003). GPT-4o with RadCoT achieved the highest F1 (0.93). Llama-3.3-70B with RadCoT (F1 = 0.89) narrowed the gap with GPT-4o under standard prompting (F1 = 0.88; paired t-test, Holm-adjusted p = 0.28). Interpretation errors showed the largest gain, with F1 improving from 0.57 to 0.75. RadCoT consistently enhances LLM-based error detection, particularly for complex logic and consistency errors, closing the gap between open-source and commercial models and offering a pathway for privacy-preserving, on-premises QA. Question Manual review misses radiology report errors and remains difficult to scale. Findings Structured prompting improves detection of interpretation and section-consistency errors across seven models. Relevance statement Open-source models approach commercial performance for privacy-preserving on-premises radiology quality assurance.
It is suggested that domain-specific training matters more than model scale for PET/CT report error detection, supporting compact models as an accurate and computationally efficient approach to automated radiology report quality assurance.
Hermione Warr, Harry Anthony, Lilli J. Freischem et al.· 0 citations
RadMatch is the most clinically aligned metric, matching inter-radiologist agreement on ReXVal and more than doubling the best prior metric on the harder RadEvalExpert, and is designed to extend to other modalities and anatomies.
Charles Corbière, Léo Machado, Aubin Charley et al.· 1 citation
This Educational Editorial & Practical Guide translates CARE-radiology, CARE, ICMJE, COPE, the 2024 Declaration of Helsinki and CRediT into a 12-step SJORANM workflow for authors, reviewers and editors to create an educational, internally consistent and independently verifiable case report before peer review.
F. Mosler, G. Nöldge, Keivan Daneshvar· Swiss Journal of Radiology a...· 0 citations
Giving radiologists EHR context through a dashboard did not change the specificity of STAT priority CXR reports or prescribing patterns, and was associated with small improvements in diagnostic concordance and follow-up CT use, without slowing reporting.
J. Balkman, S. Chimmula, Alysha Lam et al.· medRxiv· 0 citations
This paper quantifies the sensitivity of established evaluation metrics to variations in reporting practices and introduces a radiologist-informed taxonomy of variations in radiology reporting practice and a method (ReRef) that rewrites reference reports along the axes of the authors' taxonomy while preserving clinical...
Daniel P. Jeong, Charles Q. Li, H. Hosseiny et al.· arXiv.org· 0 citations
This is the first study to apply an LLM for automated classification of radiation oncology safety events and to outline a framework for future reporting systems that leverage artificial intelligence (AI)-assisted workflows, demonstrating strong potential for automating retrospective safety event tagging and streamlinin...
Qiong-Ge Li, Jian Liu, Xing Li et al.· Practical Radiation Oncology· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT scientists developed a training method to generate models that simulate the effects of human activities on climate much more quickly than other approaches.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.