MAMGAT-Net: A Multimodal Graph Attention and Multi-Scale Feature Fusion Framework for Early Breast Cancer Diagnosis from Mammography and Histopathological Images
TL;DR
A novel framework called MAMGAT-Net, which integrates the multimodal approach, Graph Attention, and Multi-Scale Feature Fusion networks for breast cancer diagnosis, is introduced, which is interpretable and clinically relevant based on the Grad-CAM and Integrated Gradients analysis.
Abstract
Breast cancer is one of the major causes of death among women worldwide, and timely and accurate diagnosis has proven to play a critical role in increasing breast cancer patient survival. Traditional single-modality diagnostic systems, however, tend to be unable to capture both the macro-level structural abnormalities that can be seen in mammograms and the micro-level cellular characteristics seen in histopathological images. To overcome this, this research introduces a novel framework called MAMGAT-Net for the diagnosis of breast cancer, which combines the multimodal approach with Graph Attention and Multi-Scale Feature Fusion networks. To address this challenge, this work proposes a novel framework, MAMGAT-Net, which integrates the multimodal approach, Graph Attention, and Multi-Scale Feature Fusion networks for breast cancer diagnosis. The proposed architecture combines the multi-scale convolutional mammography branch and ResNet-based histopathology branch with a dual-stream feature extraction mechanism. A cross-modal attention mechanism is used to learn to align and fuse complementary diagnostic information across modalities, in a bidirectional fashion. Then, a multimodal diagnostic region graph is built with a Cross-Gated Multi-Head Graph Attention Network to capture complex relational relationships among multimodal diagnostic regions, and global graph pooling and feature refinement are used to provide robust classification. Experimental results on CBIS-DDSM and BreaKHis datasets show that MAMGAT-Net can outperform the conventional machine learning and deep learning baselines with 94.59% accuracy, 92.90% precision, 96.56% recall, 94.70% F1-score, and 98.58% AUC. The effectiveness of multi-scale feature extraction, attention-based fusion, and graph relational learning is further validated in ablation studies. Furthermore, the proposed framework is interpretable and clinically relevant based on the Grad-CAM and Integrated Gradients analysis. The results show that the multimodal breast cancer diagnosis solution of MAMGAT-Net is effective and reliable.