Modal Characteristic Optimization and Spatiotemporal Correlation Features Fusion for Multimodal Disease Diagnosis

Abstract

The integration of diffusion tensor imaging (DTI) and functional magnetic resonance imaging (fMRI) for computer-aided diagnosis presents significant challenges, primarily due to their diverse spatial-temporal representations. Furthermore, current multimodal fusion techniques frequently encounter challenges in differentiating between shared and individualized information across various modalities, which restricts their capacity to identify nuanced structural and functional abnormalities linked to the progression of neurodegenerative conditions. In response to these challenges, we introduce a framework known as modal characteristic optimization and spatiotemporal correlation features fusion (MCO-SCFF) aimed at diagnosing mild cognitive impairment (MCI) and Alzheimer’s disease (AD). MCO-SCFF utilizes specialized encoders to derive both shared and modality-specific representations from DTI and fMRI data, facilitating efficient intra- and inter-modality fusion. Furthermore, a brain region spatiotemporal features aggregator (BR-SFA) is proposed to consolidate spatiotemporal features across various brain regions through the implementation of a graph-based architecture. Extensive experiments on the ADNI and PPMI datasets demonstrate the effectiveness of the proposed framework. In particular, on the ADNI dataset, MCO-SCFF achieves 84.8±2.6\% accuracy, 93.6±7.3\% sensitivity, and 82.9±6.8\% specificity on the challenging MCI vs AD task. On the PPMI dataset, it achieves 85.6±5.3\% accuracy, 86.4±4.8\% sensitivity, and 85.9±7.6\% specificity, showing robust and competitive performance for multimodal neurodegenerative disease diagnosis. Our code will be released to the community.