Skip to content

M2G-LLM: Enhancing Clinical Prediction via Multimodal Graph Reasoning and LLM Context Injection

Sep 2026 · arXiv.org · Vol abs/2609.21164 · 0 citations · 35 references
Computer Science

TL;DR

The proposed M2G-LLM (Multimodal MedGraph-LLM), a novel framework that enhances LLMs with multimodal integration and alignment via Graph Neural Networks (GNNs), highlights the promise of combining the language understanding of LLMs with the relational reasoning capabilities of GNNs for comprehensive, multimodal healthcare analysis.

Abstract

Integrating diverse data modalities --- such as clinical notes, laboratory results, and medical imaging --- is essential for advancing clinical decision-making. While Large Language Models (LLMs) have shown remarkable performance in processing unstructured clinical text, their limited capacity to incorporate non-text modalities hinders their broader utility in healthcare applications. Here, we introduce M2G-LLM (Multimodal MedGraph-LLM), a novel framework that enhances LLMs with multimodal integration and alignment via Graph Neural Networks (GNNs). Our approach models temporal relationships between patient visits, propagates information across clinically similar patients, and aligns heterogeneous data sources to construct enriched multimodal context vectors. These vectors are injected into the intermediate layers of the LLM, enabling joint reasoning over textual and non-textual modalities. We evaluate M2G-LLM on the MIMIC-IV and MIMIC-CXR datasets, demonstrating improvements in clinical prediction tasks over strong baseline models. Our results highlight the promise of combining the language understanding of LLMs with the relational reasoning capabilities of GNNs for comprehensive, multimodal healthcare analysis.

View source

Similar papers

Open access Aug 2026

CARE-LLM-GRAPH: Confidence Aware LLM integrated Multimodal Architecture for clinical Recommendation

A new confidence-aware hybrid design, CARE-LLM-GRAPH, which combines large language models (LLMs) to perform clinical reasoning, multimodal deep learning to analyze medical images, and population-aware graph intelligence to provide cohort-level information is presented.

Unknown authors · 0 citations
Review Open access Oct 2024

Large Language Model Benchmarks in Medical Tasks

A comprehensive survey of various benchmark datasets used in medical LLM tasks, focusing on various aspects of medical knowledge such as electronic health records, doctor–patient dialogues, medical question answering, and medical image captioning, is presented.

L. K. Yan, Qian Niu, Ming Li et al. · 34 citations · ⚡1
#natural language process... Preprint Aug 2026

Surgical Alignment in Knowledge Graph Training for Clinical Diagnosis with Large Language Models

A systematic study spanning five KG task formulations, three training paradigms, two KGs, and three base LLMs finds that at the task level, all paradigms improve over the non-finetuned baseline, but methods with comparable in-domain accuracy show substantially different knowledge transfer behavior.

Saksham Khatwani, He Cheng, M. Afshar et al. · 0 citations
Open access Sep 2026

Less Can Be Better: Decomposing Clinical Data Modalities in Large Language Model-based Healthcare Applications

The benefits of multimodal data integration are task-dependent and healthcare LLMs should examine clinical data modalities according to specific tasks for efficient integration, and provide practical guidance for designing efficient clinical decision support systems.

Cheng Peng, Mengxian Lyu, Ziyi Chen et al. · 0 citations
Preprint Aug 2026

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

This work introduces MedReaMM, a benchmark specifically designed to evaluate models'ability to synthesize heterogeneous clinical evidence consisting of detailed patient histories alongside multiple medical images into accurate differential diagnoses under a complete-information paradigm.

Lai Wei, Yu-Chao Chen, Zhenbiao Cao et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.