This work investigates whether image pretraining on 3D CTA scans reduces modality imbalance and improves cross-modal integration for stroke recurrence prediction, a clinically critical task recently addressed.
Abstract
Multimodal stroke recurrence prediction requires effective integration of heterogeneous clinical and imaging data, yet modality imbalance often causes models to over-rely on dominant modalities and underutilize complementary information. While self-supervised pretraining and selective parameter freezing are commonly employed to improve representation learning and fine-tuning stability, their effect on modality contributions and cross-modal behavior in multimodal medical models remains largely unexplored. In this work, we investigate whether image pretraining on 3D CTA scans reduces modality imbalance and improves cross-modal integration for stroke recurrence prediction, a clinically critical task we recently addressed. To this end, two multimodal neural networks are pretrained in a self-supervised manner and subsequently fine-tuned using two distinct freezing strategies. Their performance and modality utilization are compared against both the baseline model from our previous work and models trained entirely from scratch in this study. Our results demonstrate that self-supervised pretraining enables more effective utilization of the multimodal image-tabular dataset, outperforming both the prior baseline and all non-pretrained models. Notably, the best-performing Vision Transformer based neural network successfully overcomes unimodal collapse. Synergy analysis reveals significant interactions between vision and both gender and CHD, suggesting clinically relevant patterns for stroke recurrence. Overall, our findings demonstrate that self-supervised pretraining and strategic fine-tuning support more balanced modality utilization and enable meaningful cross-modal interactions. Code is publicly available at https://github.com/ChristianGappGit/SSL_Pretraining.
Selective Cross-Modal Learning is introduced, a lightweight pre-fusion refinement framework that enriches unimodal embeddings by selectively incorporating complementary cues from concurrently available modalities and maintains its advantage under missing-pathology and missing-genomics conditions.
Mohammed Belghadi, Younès Bennani, Valérie Paradis· 0 citations
The proposed multimodal evaluation setup examines several public datasets, offering a well-designed statistical analysis framework and research-practice reproducibility and initial results indicate representative power and proper data alignment as crucial elements.
L. V. van Dijk· American International Journ...· 0 citations
Depression manifests through heterogeneous behavioral cues, and multimodal methods can improve its recognition by integrating complementary information across modalities. However, existing approaches remain limited by the high computational cost of long-sequence modeling, insufficient cross-modal interaction, and inter...
NeuroFusion achieves state-of-the-art brain age prediction on the OpenBHB benchmark and maintains strong performance with only 5 labelled examples, demonstrating clinically relevant generalizatio in few-shot adaptation to unseen sites.
Molla Md Rony, Rahman Md Takibur, Md. Emran Hossain Dipu et al.· International Journal of Adv...· 0 citations
This work introduces Latent-Attention Masked Autoencoder (LAMAE), a multimodal, structure-aware masked autoencoder that jointly learns patient-level representations during self-supervised pretraining that outperforms modality-specific pretraining and strong contrastive and vision-language baselines across multimodal ho...
Andrea Agostini, Simon Böhi, Moritz Vandenhirtz et al.· 0 citations
Skin diseases represent a major global public health burden, yet machine learning tools developed to assist in their diagnosis suffer from two critical limitations: reliance on only one modality for diagnosis and systematic performance disparities across skin tones. While existing approaches address each challenge sepa...
Asonyu Senge Njih, Yvan Guifo Fodjo, Vianney Kengne Tchendji et al.· 0 citations
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.