Disease-Aware Visual Token Memory for Chest X-Ray Report Generation with VLMs

Abstract

Existing medical vision–language models (VLMs) for chest X-ray (CXR) report generation project all visual patch tokens into a frozen large language model (LLM) uniformly, diluting pathology-bearing evidence and biasing output towards generic, normality-dominated descriptions. We propose Disease-Aware Visual Token Memory (DA-VTM), which conditions both representation and retrieval on disease semantics. A MedCLIP-initialised Swin Transformer encodes the CXR, which a Grad-CAM guided cross-attention module distils into 14 disease-aware visual tokens, one per CheXpert finding. These key a dual Hopfield associative memory retrieving prototypical disease evidence and a clinically similar report exemplar, both passed to a frozen LLaMA-3-8B decoder via gated cross-attention adapters. On MIMIC-CXR, DA-VTM gains +2.4 CE-F1 and+1.5 RadGraph-F1 over the strongest baseline but only +0.6 BLEU-4, at ∼15% inference overhead; the gains transfer to IU X-Ray and zero-shot CheXpert Plus (+1.9 CE-F1 each). Improvements remain bounded by the fixed 1 4-label taxonomy and by laterality-agnostic retrieval.