Multimodal AI Methods for Predicting Pathologic Complete Response in Breast Cancer.
TL;DR
A novel multimodal deep learning approach that combines pre-treatment dynamic contrast-enhanced MRI and clinical data for predicting pCR before chemotherapy, and is the first to integrate attention-based multiple instance learning technique for slice aggregation and a self-supervised contrastive objective to align image and clinical embeddings is proposed.
Abstract
Neoadjuvant chemotherapy (NAC) can eliminate all invasive cancer in some breast cancer patients, achieving a pathologic complete response (pCR) that is associated with a better prognosis. Prediction of pCR from pre-treatment radiology imaging is challenging but could provide immense value in the treatment planning. We propose a novel multimodal deep learning approach that combines pre-treatment dynamic contrast-enhanced MRI and clinical data for predicting pCR before chemotherapy. It is the first to integrate attention-based multiple instance learning technique for slice aggregation and a self-supervised contrastive objective to align image and clinical embeddings. Our model was trained on 1491 patients across four cohorts with expert tumor segmentations. On a stratified hold-out test set, the multimodal model achieved AUC 0.83 (95% CI 0.78-0.89), outperforming late fusion without contrastive alignment and single-modality baselines. These results, obtained on the largest multi-cohort breast MRI dataset comprising four clinical trials, underscore the potential of contrastive learning-based multimodal AI models to improve pCR prediction. With further validation, our proposed approach could support pre-treatment decision-making by identifying patients likely to achieve pCR and those who may benefit from alternative regimens.