Skip to content
Book Open access

CGFusion: Concept-Guided Multimodal Fusion for Interpretable Cancer Diagnosis and Prognosis

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 41 references

Abstract

Integrating radiology, whole-slide pathology, and structured clinical data is essential for reliable cancer prediction, yet remains challenging due to substantial cross-modal heterogeneity. Existing multimodal models often fail to establish consistent cross-modal representations and provide limited semantic interpretability, making their predictions difficult to trust in clinical settings. To address these challenges, we propose CGFusion, an interpretable multimodal framework that jointly models representation alignment and concept-level semantic grounding. First, a patient-centric contrastive module explicitly aligns CT and WSI representations at the patient level, reducing semantic inconsistency between macroscopic radiology and microscopic pathology. Second, structured clinical variables are projected to initialize the latent queries of a Perceiver-based fusion network, enabling patient-specific priors to guide cross-modal interaction from the outset. Third, an LLM-guided concept grounding module anchors fused latent tokens to clinically meaningful prototypes through temperature-scaled similarity, token-dependent abstention, and hierarchical regularization. Experiments on NLST and MMIST ccRCC demonstrate that CGFusion achieves strong overall performance on NLST and delivers more balanced predictions than competitive baselines on MMIST under severe class imbalance. These results suggest that explicitly coupling multimodal alignment with concept-level grounding is a promising direction for building reliable and interpretable multimodal medical systems.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.