Skip to content

Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework

Sep 2026 · 0 citations · 17 references
Computer Science

TL;DR

This work investigates whether image pretraining on 3D CTA scans reduces modality imbalance and improves cross-modal integration for stroke recurrence prediction, a clinically critical task recently addressed.

Abstract

Multimodal stroke recurrence prediction requires effective integration of heterogeneous clinical and imaging data, yet modality imbalance often causes models to over-rely on dominant modalities and underutilize complementary information. While self-supervised pretraining and selective parameter freezing are commonly employed to improve representation learning and fine-tuning stability, their effect on modality contributions and cross-modal behavior in multimodal medical models remains largely unexplored. In this work, we investigate whether image pretraining on 3D CTA scans reduces modality imbalance and improves cross-modal integration for stroke recurrence prediction, a clinically critical task we recently addressed. To this end, two multimodal neural networks are pretrained in a self-supervised manner and subsequently fine-tuned using two distinct freezing strategies. Their performance and modality utilization are compared against both the baseline model from our previous work and models trained entirely from scratch in this study. Our results demonstrate that self-supervised pretraining enables more effective utilization of the multimodal image-tabular dataset, outperforming both the prior baseline and all non-pretrained models. Notably, the best-performing Vision Transformer based neural network successfully overcomes unimodal collapse. Synergy analysis reveals significant interactions between vision and both gender and CHD, suggesting clinically relevant patterns for stroke recurrence. Overall, our findings demonstrate that self-supervised pretraining and strategic fine-tuning support more balanced modality utilization and enable meaningful cross-modal interactions. Code is publicly available at https://github.com/ChristianGappGit/SSL_Pretraining.

View source

Similar papers

Selective Cross-modal Feature Enhancement Through Collaborative Learning

Selective Cross-Modal Learning is introduced, a lightweight pre-fusion refinement framework that enriches unimodal embeddings by selectively incorporating complementary cues from concurrently available modalities and maintains its advantage under missing-pathology and missing-genomics conditions.

Mohammed Belghadi, Younès Bennani, Valérie Paradis · 0 citations
Open access 2024

Deep Neural Frameworks for Integrating Multimodal Healthcare Data

The proposed multimodal evaluation setup examines several public datasets, offering a well-designed statistical analysis framework and research-practice reproducibility and initial results indicate representative power and proper data alignment as crucial elements.

L. V. van Dijk · 0 citations
Open access 2026

SPLGMMamba: A Self-Paced Learning-Guided Multimodal Mamba Framework for Depression Recognition

Depression manifests through heterogeneous behavioral cues, and multimodal methods can improve its recognition by integrating complementary information across modalities. However, existing approaches remain limited by the high computational cost of long-sequence modeling, insufficient cross-modal interaction, and inter...

Ze-Feng Shi, Hui-Min Mi, Jing-Yi Liu · 0 citations
Open access 2026

Brain Age Prediction via Self-Supervised Multitask Learning with Cross-Modal Fusion

NeuroFusion achieves state-of-the-art brain age prediction on the OpenBHB benchmark and maintains strong performance with only 5 labelled examples, demonstrating clinically relevant generalizatio in few-shot adaptation to unseen sites.

Molla Md Rony, Rahman Md Takibur, Md. Emran Hossain Dipu et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning

This work introduces Latent-Attention Masked Autoencoder (LAMAE), a multimodal, structure-aware masked autoencoder that jointly learns patient-level representations during self-supervised pretraining that outperforms modality-specific pretraining and strong contrastive and vision-language baselines across multimodal ho...

Andrea Agostini, Simon Böhi, Moritz Vandenhirtz et al. · 0 citations
Preprint Aug 2026

MIFR: A Modality-Invariant and Fair Representation Framework for Skin Disease Classification

Skin diseases represent a major global public health burden, yet machine learning tools developed to assist in their diagnosis suffer from two critical limitations: reliance on only one modality for diagnosis and systematic performance disparities across skin tones. While existing approaches address each challenge sepa...

Asonyu Senge Njih, Yvan Guifo Fodjo, Vianney Kengne Tchendji et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.