Skip to content
Open access

Multimodal graph neural networks for integrating heterogeneous medical data in clinical decision making

Aug 2026 · Discover Computing · Vol 29 · 0 citations · 39 references

TL;DR

A multi-modal Graph Neural Network model, DUA-HIR, is proposed, which combines structured and unstructured information, such as text and images, to enhance diagnosis and risk prediction.

Abstract

Healthcare data is complex, with Electronic Health Records (EHRs), clinical images, notes, and genetic data, challenging and enabling the development of effective clinical decision support systems. In this paper, we propose a multi-modal Graph Neural Network (GNN) model, DUA-HIR, which combines structured and unstructured information, such as text and images, to enhance diagnosis and risk prediction. It leverages GNNs to model patient relationships, Convolutional Neural Networks (CNNs) to process images, and Transformers to analyse text. It uses attention and multi-task learning for improved predictions of clinical outcomes. The model is thoroughly evaluated on MIMIC-III and MIMIC-CXR, showing substantial improvements in a range of key clinical prediction tasks. Ablation experiments, sensitivity analyses, calibration measures and computational costs confirm the model’s architecture and practicality. The paper also explores ethical implications for clinical use.

Read PDF

Similar papers

Conference Jul 2026

Temporal Graph Neural Network and LSTM-based Hybrid Model for Early Cancer Prediction in Connected Healthcare Systems

Cancer continues to be a significant global health concern, inflicting considerable financial and healthcare burdens on individuals, healthcare providers and national economies. Traditional diagnostic methods frequently do not identify tumours at an early and manageable stage, resulting in heightened treatment complexity and healthcare expenses. In this context, Early Cancer Prediction within Connected Healthcare Systems has become a significant research domain, utilising big data and predictive analytics to enhance healthcare outcomes. The suggested framework amalgamates diverse healthcare data sources, such as electronic health records, genomic databases, medical image repositories, wearable devices, and insurance claims, to construct an intelligent predictive model. Data preprocessing employs the StandardScaler technique for dataset normalisation, while recursive feature selection is utilised to discern the most pertinent features from the cancer prediction dataset, hence improving classification performance. A deep learning architecture based on LSTM is utilised for prediction, using a diffusion convolution kernel to capture delay propagation features, culminating in the DeepGraphLSTM model. Experimental findings indicate that the suggested model surpasses current mainstream methodologies for robustness and prediction accuracy, with an accuracy of 94.08%. The study comes to the conclusion that combining deep learning with big data analytics greatly improves early cancer detection and facilitates effective decision-making in networked healthcare settings.

Kumar Mohan, Mohamed Kuresh, Amina Salim Mohammed Al Jabri et al. · 0 citations
Open access Aug 2026

Chronic Disease Prediction Using Temporal Convolutional and Attention-Guided LSTM Networks

The early prediction of chronic illnesses such as diabetes, hypertension, and heart failure is vital for improving patient outcomes and reducing healthcare costs. However, Electronic Health Records (EHRs) present challenges due to their heterogeneous nature, containing a mix of structured (e.g., laboratory values, vital signs), unstructured (e.g., clinical notes), and temporal data (e.g., patient history). This paper proposes a novel deep ensemble learning framework that integrates Temporal Convolutional Networks (TCNs) and attention-guided Long Short-Term Memory (LSTM) networks for robust chronic disease prediction. The TCN module captures long-range temporal patterns and clinical trends across patient timelines, while the attention-enhanced LSTM module focuses on relevant time-steps and feature segments critical for diagnosis. The ensemble approach combines these deep representations to improve classification accuracy, reduce false positives, and provide resilience against data irregularities. Experiments conducted on benchmark multi-modal EHR datasets demonstrate the model’s superiority over traditional RNN, LSTM, and CNN-based methods. The proposed architecture not only enhances prediction performance but also contributes toward explainable and proactive healthcare decision-making.

T. Thamaraiselvan, K. Saravanan, S. Nithyanandam · 0 citations
Conference Jul 2026

Integrating Explainable AI and Graph Neural Networks for Disease Prediction: A Comprehensive Review of EHR-based Healthcare Models

Due to Artificial Intelligence (AI) and Machine Learning (ML), the Healthcare sector has increased rapidly and progressed in disease prediction, diagnosis and chronic disease management. But these technologies have not been fully utilized because of many advanced models as they are difficult to understand. And to overcome all this, Explainable Artificial Intelligence (XAI) has been developed. That makes the model easy to understand and makes decisions. Also, help to increase the trust of doctors and patients. Addition to this, the Graph Neural (GNNs) and Graph Convolutional Networks (GCNs) such as graph-based methods, Electronic Health Records (EHR) have complex relational ships for understanding them better. In this review of 2024-2026 the 20 studies have been covered. That contains heart disease, diabetes, cancer and mental health, etc. SHAP, LIME, and Attention mechanism techniques make explainability better. Whereas, due to lack of data, real world validations and high computational cost remain.

Divyanshu Divyanshu, Prateek Srivastava · 0 citations
Review Open access Aug 2026

Deep learning and generative AI for medical imaging and clinical decision support systems: a structured critical review

It is argued that LLM-based CDSS are not supported for routine autonomous use and require clinician supervision, and set out a research agenda centred on validation, governance and human-in-the-loop deployment.

A. Mohan Babu, A. Jeshurun Nehemiah, V. Jagadeep et al. · 0 citations
Jul 2026

Enhancing Transparency in Cardiovascular Disease Prediction Using CNN-MLP and Explainable AI Framework for Sustainable Health Care

Cardiovascular disease (CVD) requires more specific and comprehensible diagnostic tools, as it causes numerous deaths in the world. This study aims to provide a multimodal, comprehensive, effective, and clinically interpretable AI model for heart disease prediction. Sometimes medical practitioners may find it difficult to comprehend and trust the model's decision-making process, and the “black box” nature of these models limits their practical usefulness despite their high predictive accuracy. To address this challenge, the ongoing research proposes an explainable AI (XAI) framework that integrates a Multi-Layer Perceptron (MLP) and a two-dimensional Convolutional Neural Network (2D-CNN) with Layer-wise Relevance Propagation (LRP) for transparent, potent, and reliable cardiac disorder prediction. LRP reveals the most significant features for interpreting the model's output by assigning relevance scores to each input feature that leads to a particular prediction. Local and global explanations are generated to assist the stakeholders in understanding the rationale for the decision. The model achieves 98% accuracy on Electrocardiogram (ECG) data and 97% on Electronic Health Record (EHR) data, indicating strong generalization and high accuracy. While the proposed CNN-MLP with LRP framework achieved high performance and interpretability, several limitations must be acknowledged. To start with, the research was based on the publicly available dataset of ECG images and EHR data that might not be a complete reflection of the diversity of the real-world clinical populations. Consequently, the model has an unclear ability to generalize information. Second, the dataset size, though sufficient for proof-of concept, may still be limited for deep learning applications, and the lack of an independent test set could slightly overestimate performance. Third, the model was trained on static ECG images rather than raw time-series signals, potentially omitting temporal dependencies critical for certain cardiac diagnoses. Furthermore, Layer-wise Relevance Propagation (LRP) explanations are informative but require model architecture and can differ among implementations. Finally, the study was performed in a controlled, offline environment; in practice, it would take a lot of clinical validation, hospital systems integration, and evaluation of computational efficiency and interpretability. This research proposes a productive, interpretable diagnostic tool for cardiac disease that integrates Electrocardiogram (ECG) and Electronic Health Record (EHR) data, identifying the most informative features by assigning relevance scores. The CNN-MLP and LRP framework efficiently fuses diverse data for the early diagnosis of cardiovascular diseases in an understandable and privacy-aware manner. It improves clinical decision-making by providing relevance scores for multimodal features, thereby facilitating trustworthy AI-assisted diagnosis in cardiology.

Sima Ghosh, Bihan Chowdhuri, Soumyadeep Mondal et al. · 0 citations