Adaptive Confidence-weighted Expansion (ACE), a novel framework to enhance the trustworthiness of multimodal fusion models, provides a more stable and robust data fusion method that facilitates the use of multimodal learning in addressing high-stakes problems.
Abstract
Multimodal learning is a robust approach to improve predictive performance in applications such as medical prognosis. However, the clinical applicability of models that use multimodal learning is hampered by their poor performance under noisy or uninformative data streams. Present fusion approaches often lack robust mechanisms for the dynamic assessment of data quality and for the provision of a trustable confidence score on the final prediction. This dissuades their deployment in safety-critical settings. To address these limitations, we introduce Adaptive Confidence-weighted Expansion (ACE), a novel framework to enhance the trustworthiness of multimodal fusion models. ACE first enhances the multimodal space by generating new, complementary modalities from intra-modality correlations. It then employs a dual-level confidence mechanism that (1) adaptively reweighs all modalities by their reliability before fusion and (2) estimates a global trust score over the fused, final decision. To evaluate ACE, we used four challenging multi-omics datasets (BRCA, KIPAN, LGG, and ROSMAP). ACE significantly outperforms existing state-of-the-art algorithms in both classification performance and confidence calibration. Our framework provides a more stable and robust data fusion method that facilitates the use of multimodal learning in addressing high-stakes problems.
Background: Reliability-aware integration of heterogeneous data sources remains a fundamental challenge in multimodal deep learning: prevailing fusion strategies assume uniform reliability across sources and instances, limiting their responsiveness to data-dependent trustworthiness. Methods: We introduce REM-Fuse (Reliability-aware Evidential Multimodal Fusion), an evidential deep learning (EDL) framework in which per-source Dirichlet uncertainty adaptively weights each source through dual-channel weighting, asymmetric cross-scale enhancement, and Dempster–Shafer-inspired evidence accumulation. As a case study for cancer prognosis, REM-Fuse integrates multi-scale histopathology (10×, 20×) and RNA-seq on TCGA-BRCA (n = 831) via five-fold cross-validation with subtype- and stage-stratified analyses. Results: REM-Fuse attained a concordance index of 0.715 and a 60-month time-dependent AUC of 0.729, indicating moderate discrimination and significant risk separation (log-rank p < 0.001). Adaptive source weights and per-patient uncertainty varied significantly across molecular subtypes (Kruskal–Wallis p = 0.010 and p = 0.007), indicating patient-specific rather than fixed multimodal integration. Conclusions: REM-Fuse provides a compact reliability-aware fusion strategy for cancer prognosis, although external validation is needed before broader clinical or cross-cohort generalization.
Yalu Huang, Yu-Shuai Yuan, Wenbin Ye et al.· Mathematics· 0 citations
The promise of multimodal fusion lies in combining complementary sources of evidence, yet more evidence does not always yield a better prediction. Recent multimodal models have advanced fusion through richer cross-modal interaction and sample-adaptive fusion. However, the influence assigned to a modality during fusion does not reveal whether that source is unreliable, redundant, or poorly matched to a specialized expert. To address this limitation, we introduce TIER-MoE, a risk-guided subspace mixture-of-experts model that defines sample-specific modality reliability as the prediction loss its unimodal predictor is expected to incur. This risk is learned from out-of-fold predictions generated by models that were not trained on the corresponding sample. TIER-MoE combines the estimated risk with expert-specific subspace compatibility for sparse modality-expert routing, while an always-active shared path preserves multimodal complementarity. We evaluate TIER-MoE on four public multimodal biomedical datasets spanning Alzheimer's disease status, skin-lesion malignancy, and retinal classification. Results demonstrate its superiority over state-of-the-art methods in predictive performance and probability calibration, with consistent improvements in Macro-F1 and Brier score and strong zero-shot generalization to an external cohort.
Yung-Chun Chang, Anzhe Cheng, Chenwei Wu et al.· 0 citations
The growing availability of heterogeneous clinical data has provided new prospects of data-driven clinical decision support, but at the same time, brought about challenges to do with multimodal integration, uncertainty management, and interpretability. Current AI systems in clinical settings are also likely to be based on fixed fusion methods and single-mode thinking, which restricts their resilience and reliability in practice in healthcare settings. In order to overcome these shortcomings, this paper presents a new confidence-aware hybrid design, CARE-LLM-GRAPH, which combines large language models (LLMs) to perform clinical reasoning, multimodal deep learning to analyze medical images, and population-aware graph intelligence to provide cohort-level information. The new framework clearly formulates modality-specific uncertainty and also uses a confidencesensitive adaptive fusion process to combine dynamically the text, visual and graph-based evidence. In addition, the iterative refinement process that is organized by an LLM allows an adaptive reasoning in situations when clinical conditions are ambiguous or incomplete. Most experiments performed out of publicly available multimodal clinical datasets show that CARE-LLM-GRAPH has been demonstrating consistently better discriminative performance, recommendation ranking quality, calibration, and performance under missing data conditions compared to state-of-the-art unimodal and multimodal baselines. The framework also produces interpretable evidence-based explanations, which increase the level of clinical transparency and trust. These findings underscore how CARE-LLMGRAPH can be a valid and explicable clinical decision support framework to serve practical purposes in healthcare environments.
Unknown authors· European Journal of Prosthod...· 0 citations
Multimodal fusion learning (MFL) (a framework to jointly learn from heterogeneous data sources) has shown great potential in various fields such as Medicine, Science, and Engineering. It is extremely desirable in the medical domain, where we are faced with disparate data modalities such as imaging, clinical records, and omics. However, existing MFL strategies face several major challenges. First, they struggle to capture complex cross-modal interactions effectively, which in turn limits performance improvements. Second, they incur high computational costs, restricting their applicability in resource-constrained healthcare AI applications. Finally, they are often designed and evaluated for narrow, fixed modality configurations (e.g., imaging-only, or specific pairs such as image and omics), which limits evidence of their adaptability and generalizability to broader collections of heterogeneous medical modalities. To address these challenges, we propose a novel MFL framework – Cascaded Unified Representation Learning for Efficient Fusion Network (CURE) – a lightweight and scalable framework that progressively integrates various modalities through a novel efficient Hybrid Geometry Aware Fusion layer (HyFuse), where each HyFuse layer is sequentially learned for each modality, making the framework adaptable and generalizable. Within HyFuse, an efficient residual convolution module captures rich multi-scale features to ensure cost-effective learning, while a hybrid-space aware attention mixer learns coarse-to-fine structural cues to better preserve cross-modal relationships. Complementary learnable late-fusion and shared-information refinement modules are then employed to learn robust, modality-order-invariant shared features, which in turn yields consistent performance improvements. Extensive evaluations on 16 public datasets show that CURE outperforms leading multimodal fusion methods (e.g., DRIFA-Net and HEALNet), boosting performance by up to ≈ 3.97% and lowering computational costs by up to ≈ 87.8%, ensuring more effective and reliable predictions.
Joy Dhar, M. Pandey, Nayyar Zaidi et al.· Proceedings of the 32nd ACM...· 0 citations
Multi-omics data integration holds great promise for precision medicine, yet its clinical adoption is hindered by high acquisition costs and the complexity of heterogeneous data representations. To address these challenges, we propose an uncertainty-aware multi-view dynamic decision framework for efficient and trustworthy disease classification. Unlike conventional static fusion strategies, our approach leverages evidential deep learning grounded in Dempster-Shafer theory to explicitly disentangle predictive confidence from epistemic uncertainty, enabling cost-sensitive and progressive inference. Specifically, omics modalities are introduced adaptively, such that additional data are only acquired when the current evidence is insufficient to support a reliable decision. At the core of the UMCA-Net, a Transformer-based multi-stream architecture with global joint cross-attention captures rich cross-modal interactions and produces Dirichlet-based evidential representations. This design allows principled uncertainty quantification and supports dynamic decision-making. We evaluate the proposed method on four benchmark multi-omics datasets (ROSMAP, LGG, BRCA, and KIPAN). Experimental results demonstrate that our model achieves state-of-the-art performance while significantly reducing data acquisition requirements. Notably, in certain cohorts, over 90% of samples can be confidently classified using only low-cost initial modalities without compromising accuracy. Overall, this work provides a scalable and practical solution for balancing diagnostic accuracy and economic cost, facilitating the deployment of multi-omics models in real-world clinical settings. Our code is available to the public at github.com/chenzhao2023/UMCA-Net.
Yehong Huang, Huan Huang, Selena He et al.· Computers in Biology and Med...· 0 citations
Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.
Nikkie Hooman, Zhongjie Wu, Eric C. Larson et al.· 0 citations