Skip to content
#federated learning Review Open access

Transformer-Based Multimodal Image Fusion in Healthcare: A Comprehensive Survey of Architectures, Fusion Strategies, and Clinical Applications

Sep 2026 · Journal of Data Science and Intelligent Systems · 0 citations · 63 references
Advanced Image Fusion Techniques

TL;DR

This survey organizes transformer-based fusion models for healthcare along four axes:modality, pairing, fusion level, and attention and architecture design, to give researchers and clinicians a structured entry point into this area.

Abstract

Transformer architectures have changed how multimodal medical image fusion works. Convolutional neural network (CNN)-based methods are constrained to local receptive fields; transformers use self-attention and cross-modal interactions to capture dependencies across the full image, which matters when aligning magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), and ultrasound—modalities that differ substantially in resolution, contrast, and the anatomical features they emphasize. This survey organizes transformer-based fusion models for healthcare along four axes:modality, pairing, fusion level, and attention and architecture design. Four representative models—MACTFusion, ECFusion, MATR, and DFENet—are examined in detail, with attention to how each handles anatomical preservation,semantic reasoning, and clinical interpretability. Fusion strategies fall into pixel-level, feature-level, segmentation, and neurodegeneration staging, and the survey covers the evaluation metrics and benchmark datasets used to assess performance across these tasks. A number of problems have not been resolved. Data scarcity and modality heterogeneity continue to limit generalization; interpretability remains shallow in most deployed architectures; and clinical environments impose constraints—compute budgets, latency requirements, and regulatory oversight—that most published models are not designed for. Work on lightweight vision transformers, language-guided prompting, 3D and temporal extensions, and federated learning addresses some of these gaps, though none yet offers a comprehensive solution. The survey aims to give researchers and clinicians a structured entry point into this area.   Received: 20 November 2025 | Revised: 13 July 2026 | Accepted: 28 August 2026   Conflicts of Interest The authors declare that they have no conflicts of interest in this work.   Data Availability Statement Data sharing is not applicable to this article, as no new data were created or analyzed in this study. The benchmark datasets discussed in this survey (BRATS, ADNI, CHAOS, OASIS, ISLES, TCIA, IXI, and EyePACS) are third-party resources, described in their original publications and listed in Section 5.1.1 and Table 8. Most are publicly accessible; a few, including ADNI and TCIA, require researchers to register with the data provider and sign a data use agreement before access is granted. Access to all datasets is subject to the terms set by their respective providers; readers should consult the original sources cited here for current access procedures.   Author Contribution Statement Nalini S Jagtap: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing – original draft, Writing – review & editing, Visualization. Dilip Kumar Jang Bahadur Saini: Conceptualization, Methodology, Software, Validation, Formal analysis, Resources, Writing – review & editing, Supervision. Prasun Chakrabarti: Conceptualization, Methodology, Formal analysis, Writing –review & editing, Supervision.

Read PDF

Similar papers

Open access Aug 2026

Hybrid CNN-Transformer Framework for Multi-Disease Detection from Medical Imaging Data

The proposed framework is intended to support clinical image assessment and prioritization rather than replace expert diagnosis, and demonstrates the potential of hybrid CNN-Transformer architectures for robust and scalable computer-assisted multi-disease screening from medical imaging data.

M. Balakrishnan, K. Ananthi, S. R. et al. · 0 citations
Open access Sep 2026

MedFuse: dual-stream fusion of convolutional and vision transformer-based features for enhanced medical image classification

Introduction Medical image classification is fundamental to computer-aided diagnosis. Limited labeled samples, subtle inter-class differences, and heterogeneous lesion morphology make it difficult for a single representation to capture all relevant cues. This study investigates MedFuse, a simple dual-stream framework t...

Ya-Jing Ren, Hai Ling, Zheng Gu et al. · 0 citations
Open access Oct 2026

A Comprehensive Framework for the Fusion of Anatomical (CT, MRI) and Functional (PET, SPECT) Medical Images for Enhanced Diagnostic Precision

Medical image fusion aims to integrate complementary information from multimodal imaging techniques to provide a more comprehensive visualization for diagnosis and treatment planning. This study presents a novel framework for fusing anatomical images (CT and MRI) with functional images (PET and SPECT), addressing chall...

P. Srivastava, Himanshu Shukla, Ravi Kant Mishra et al. · 0 citations
Open access Aug 2026

TransCat: a hybrid CNN-transformer network with KAN for medical image segmentation

TransCat, a hybrid CNN-Transformer architecture for medical image segmentation, is proposed and an extended deformable attention mechanism with attentive value identification is developed, to control the computational burden caused by the enlarged token set.

Jin Wang, Zheng-Hua Yang, Dong-Ming Zhou et al. · 0 citations
Open access Sep 2026

Computer Vision System and Transformer-Based Multimodal Fusion of Medical Images for Enhanced Diagnostic Decision Making

The proposed Multimodal Adaptive Fusion Transformer (CMAFT) is a unified architecture that combines Swin transformer encoders for each media with a Catalyst Cross Attention Fusion Module (GCAF) to dynamically weight each media's contribution based on its real-time reliability.

Soltan Abdullah, Kahlan Aljobory · 0 citations
Sep 2026

Lightweight Vision Transformer-Based U-Net for Brain Tumor Segmentation from MRI

A lightweight Vision Transformer UNet is proposed that combines the hierarchical feature extraction capability of UNet with the global context modeling of Vision Transformers, enabling effective learning of both local and global features while maintaining computational efficiency with only 2.6 million trainable paramet...

Sheekar Banerjee, Monika Chowdhury, M. Akash et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.