Enhanced vision language framework for medical report generation using agent-based retrieval

Abstract

Automated chest X-ray report generation has the potential to reduce the workload of radiologists by combining deep learning and natural language processing. However, existing vision-language models (VLMs), including R2Gen, may generate incorrect or unsupported statements when relevant contextual information is insufficiently captured. This study proposes a modular framework to improve automated report generation through inference-time retrieval and language model-based refinement. Given an input chest X-ray image, R2Gen generates a preliminary report. Relevant contextual information is retrieved using image-derived and textual representations through a FAISS-based similarity retrieval mechanism. The retrieved textual information is combined with the preliminary report and provided to the Zephyr 7B language model to generate the final report. The framework is evaluated on the IU X-Ray and CheXpert Plus datasets using BLEU, ROUGE-L, METEOR, and token-overlap-based precision, recall, and F1-score. The retrieval and refinement stages operate at inference time without requiring additional retraining or fine-tuning of the underlying R2Gen model. Experimental results demonstrate improved performance over the R2Gen baseline across the reported evaluation metrics on the evaluated datasets, indicating greater similarity between the generated reports and the reference reports. The findings suggest that integrating similarity-based retrieval with language model-based refinement can improve the reported text-generation performance of automated chest X-ray reporting systems. The modular framework offers an inference-time approach that avoids additional training of the underlying report generation model.