MedXAIgnosis: a metadata-enhanced graph-fusion pilot for thorax disease classification
TL;DR
MedXAIgnosis introduces a multimodal framework that combines chest radiographs and available clinical data to reorganize image features without extra clinician typing, and Gradient-weighted Class Activation Mapping and GNNExplainer are implemented.
Abstract
Classifying medical images is essential for the diagnosis of thorax diseases, often aided by deep learning (DL) techniques. However, traditional DL approaches typically focus on a single image type input and overlook valuable insights from clinical header data. To address this, MedXAIgnosis introduces a multimodal framework that combines chest radiograph images and available clinical data (age, sex, view-position) to reorganize image features without extra clinician typing. The framework utilizes DenseNet to extract features from chest radiographs and a Multilayer Perceptron (MLP) to map clinical header data fields into a compatible space. These features are fused and passed to a Graph Neural Network (GNN) equipped with a learnable edge threshold that prunes noisy relations on-the-fly. On the original NIH and CheXpert test sets, the plug-in reaches an AUC of 0.8078 and an accuracy of 0.8425 (NIH) and an AUC of 0.7891 with an accuracy of 0.7997 (CheXpert), which is within the same range as other metadata-only models. For model transparency, explainable AI (XAI) techniques Gradient-weighted Class Activation Mapping (Grad-CAM) and GNNExplainer are implemented. Grad-CAM highlights the most relevant pixel regions in chest X-ray images, while GNNExplainer identifies influential subgraph patterns in the GNN, outputting a compact explanation that clinicians can inspect at a glance.