Lightweight Visual Question Answering in Radiology Domain

Abstract

The increasing volume of imaging studies has motivated research on radiology Visual Question Answering (VQA), where systems combine image understanding and natural language processing to answer questions about medical images. This paper proposes a lightweight radiology VQA prototype based on MobileNetV3-Small and DistilBERT, connected through an attention-based fusion module and a dual-head classifier for binary and open-ended questions. The system is trained and evaluated on VQA-RAD, a clinician-curated benchmark of 315 radiological images and 2,248 question-answer pairs that span X-ray, CT, and MRI modalities. Using the official VQA-RAD test split, the lightweight model achieves an overall accuracy of 69.4%, compared with 66.3% for a ResNet-34 with BERT-base baseline, while reducing model size by 47% (269.59 MB vs. 511.69 MB). The measured mean inference time improves modestly from 16.357 ms to 15.751 ms on the tested device, indicating that the principal efficiency gain is model compactness rather than latency. The lightweight model also achieves higher recall on binary questions (78.81% vs. 65.25%), although these results remain preliminary because VQA-RAD is small and no external clinical validation is performed. A public Streamlit research demo and GitHub repository are provided to support reproducibility.