Interpretable Multimodal AI for Horizon-Specific Survival Prediction in Hepatocellular Carcinoma Using Transcriptomics and Caption-Based Histopathology
TL;DR
These findings show that multimodal benefit is horizon-dependent and depends on modality complementarity rather than simply adding more data sources, and that multimodal benefit is horizon-dependent and depends on modality complementarity rather than simply adding more data sources.
Abstract
Accurate survival prediction in Liver Hepatocellular Carcinoma (LIHC) remains challenging because outcomes are shaped by clinical, molecular, and morphological heterogeneity. Although multimodal learning may improve prognosis, the contribution of each modality across clinically relevant survival horizons remains unclear, and many existing approaches provide limited interpretability. We present an interpretable multimodal framework for LIHC survival prediction using clinical variables, transcriptomic profiles, and histopathology whole-slide images from the TCGA-LIHC cohort. Survival prediction was formulated as horizon-specific classification at 1-, 3-, and 5-year endpoints. Histopathology patches were converted into morphology-focused textual descriptions using a vision–language captioning framework, aggregated into patient-level summaries, and embedded as predictive features. Caption quality was evaluated using automated semantic metrics and expert pathological review. Multimodal fusion was performed using a leakage-aware out-of-fold stacking strategy. The best fusion configurations achieved ROC-AUC values of 0.70, 0.74, and 0.66 for 1-, 3-, and 5-year prediction, respectively. Genomic features provided the strongest standalone signal at 1 and 3 years, while histopathology-derived captions contributed most clearly when combined with genomics at 3 years. These findings show that multimodal benefit is horizon-dependent and depends on modality complementarity rather than simply adding more data sources.