MSPE-DeiT: A Vision Transformer-Based Multimodal Framework for Skin Cancer Diagnosis

Abstract

Skin cancer is one of the most common forms of cancer globally, with melanoma being the most fatal form. Early and precise detection is crucial for improving treatment outcomes, as timely disease management dramatically increases survival rates. This study presents a reliable multimodal deep learning framework that integrates dermoscopic images, 3D Total Body Photography scans, and patient clinical data to enhance diagnostic accuracy in skin cancer classification. The dataset comprised a combination of publicly available sources, including the recently released ISIC 2024 dataset. An XGBoost algorithm is utilized to get the predictions from the clinical data. For image-based analysis, this study proposes a Modality-Specific Patch Embedding Data efficient image Transformer (MSPE-DeiT) that uses a modality-specific patch embedding layer to train on the dermoscopic and 3D-TBP images. MSPE-DeiT is capable of capturing both local and global contextual information from high-resolution lesion images. Furthermore, this work uses an Artificial Neural Network (ANN) architecture for clinical data classification. Bayesian optimization is applied to fine-tune the ANN’s hyperparameters, effectively mitigating vanishing gradient issues and improving model convergence. A Soft-Voting Ensemble classifier is used to integrate the results of all three models for better generalization on the unseen data. Experimental results demonstrate improvements in diagnostic accuracy on a multimodal held-out test set of around 80k images from the ISIC 2024 dataset, achieving an accuracy of 91.76%, an Area Under the Receiver Operating Characteristic Curve of 0.953, a specificity of 91.77%, and a sensitivity of 84.81%. The proposed approach is computationally inexpensive with an inference latency of 6.2 ms and a memory footprint of 83.59 MB, making it fit for deployment in clinical scenarios.