Patient-specific multimodal learning with multi-view contrastive alignment for chest X-ray report generation

Abstract

Abstract Motivation Radiology reports play a pivotal role in guiding treatment planning and enabling effective doctor-patient communication. However, their manual composition imposes a substantial workload on radiologists. Although automatic radiology report generation has emerged as a promising alternative, existing approaches predominantly rely on single-view chest X-rays and fail to adequately leverage patient-specific context, thereby limiting diagnostic accuracy. Results To address this challenge, we propose EVOKE, a novel chest X-ray report generation framework that incorporates multi-view contrastive learning and patient-specific knowledge. Specifically, we introduce a multi-view contrastive learning method that captures semantic correspondences both among multi-view radiographs within a study and between these radiographs and their associated report, thereby improving visual representation learning. We further present a knowledge-guided report generation module that integrates available patient-specific knowledge (i.e. indication, which includes symptom descriptions) to facilitate the generation of accurate and coherent radiology reports. To support research in multi-view report generation, we construct Multi-view CXR and Two-view CXR datasets using publicly available sources. Our proposed EVOKE surpasses recent state-of-the-art methods across multiple datasets, achieving a 2.9% F1 RadGraph improvement on MIMIC-CXR, a 5.0% BLEU-1 improvement on MIMIC-ABN, a 1.5% BLEU-4 improvement on Multi-view CXR, and an 8.2% F1,mic-14 CheXbert improvement on Two-view CXR. Availability Code is publicly available at https://github.com/mk-runner/EVOKE, with an archived release available on Zenodo (doi:10.5281/zenodo.21000219).