GPC-CF: A Graph Convolutional Pre-Trained Contrastive Cross-Fusion Framework for Multimodal Brain Representation Learning

Abstract

By combining complementary spatiotemporal information from functional magnetic resonance imaging (fMRI) and electroencephalography (EEG), brain decoders can characterize complex multimodal neuroimaging patterns. However, prevailing unimodal approaches fail to capture multidimensional brain dynamics due to spatiotemporal resolution limitations, hindering a comprehensive understanding of brain mechanisms. To overcome these challenges, we propose a novel Graph Convolutional Pre-trained Contrastive Cross-Fusion (GPC-CF) framework, which integrates EEG and fMRI data within a pre-training and fine-tuning model. First, EEG signals are reconstructed in source space so that EEG and fMRI data share a common spatial coordinate system. Secondly, we propose a pre-training model that incorporates a cross-modal self-supervised loss function and graph edge augmentation into a contrastive learning framework to minimize inter-modal feature discrepancies. Finally, we develop a fine-tuning model integrating a cross-modal attention mechanism to facilitate mutual information exchange between EEG and fMRI, thereby learning more robust common representations. GPC-CF achieves state-of-the-art performance across five downstream tasks (major depressive disorder (MDD), anxiety disorders (AD), specific learning disorder with reading impairment (SLD), autism spectrum disorder (ASD), and attention-deficit/hyperactivity disorder (ADHD)) on the HBN dataset, with accuracies ranging from 78.8% to 88.4%. Extensive ablation studies further confirm the effectiveness and robustness of its core components. GPC-CF establishes a powerful paradigm for multimodal brain signal fusion, demonstrating consistent performance across five diagnostic tasks within HBN.