CGFusion: Concept-Guided Multimodal Fusion for Interpretable Cancer Diagnosis and Prognosis
Abstract
Integrating radiology, whole-slide pathology, and structured clinical data is essential for reliable cancer prediction, yet remains challenging due to substantial cross-modal heterogeneity. Existing multimodal models often fail to establish consistent cross-modal representations and provide limited semantic interpretability, making their predictions difficult to trust in clinical settings. To address these challenges, we propose CGFusion, an interpretable multimodal framework that jointly models representation alignment and concept-level semantic grounding. First, a patient-centric contrastive module explicitly aligns CT and WSI representations at the patient level, reducing semantic inconsistency between macroscopic radiology and microscopic pathology. Second, structured clinical variables are projected to initialize the latent queries of a Perceiver-based fusion network, enabling patient-specific priors to guide cross-modal interaction from the outset. Third, an LLM-guided concept grounding module anchors fused latent tokens to clinically meaningful prototypes through temperature-scaled similarity, token-dependent abstention, and hierarchical regularization. Experiments on NLST and MMIST ccRCC demonstrate that CGFusion achieves strong overall performance on NLST and delivers more balanced predictions than competitive baselines on MMIST under severe class imbalance. These results suggest that explicitly coupling multimodal alignment with concept-level grounding is a promising direction for building reliable and interpretable multimodal medical systems.