Optimized Deep Learning for Feature Extraction and Detection of Oral Cancer from Histopathological Images

Abstract

This study aims to develop an efficient, powerful deep learning model that accurately classifies oral cancer from high-resolution histopathology images, especially in cases with high intra-class variability and extreme class imbalance. We propose a novel Ensemble Deep Learning framework combined with two streams: CNN1(EfficientNet-B5), CNN2 (DenseNet-201), and a new Hybrid -Spatial-Channel Attention module to extract and fuse the cytological and structural features. We addressed class imbalance using a cost-sensitive learning approach and data augmentation techniques. The Whale Optimization Algorithm (WOA) was used for optimizing hyperparameters. The model is based on late-fusion of CNN1 (fine-grained features) and CNN2 (architectural features). The procedure that was followed was to utilize seven classes of 12425 high-resolution images from (ORCHID and UFES-NDB) and diagnose the seven specific histopathological classes into three primary diagnostic groups: normal, pre-cancer, and cancer. The dataset was divided into Training (80%), Validation (10%), and Testing (10%) to prevent data leakage. It is found that the overall accuracy of the proposed architecture is very robust at 90.35% on the test set. Using five-fold cross-validation. The model achieved the highest Matthews Correlation Coefficient (MCC), particularly in highly imbalanced classes. The proposed method can effectively improve the representation of features and the precision of classification for complex medical image analysis by achieving high metrics in Accuracy, Precision, Recall, Specificity, and F1-Score, which will be a powerful computational tool for automated histopathology analysis. Optimized ensemble learning enhances the detection and classification of Oral Squamous Cell Carcinoma (OSCC) and precancerous lesions by merging fine-grained features and architectural features. Future models should be validated in independent patient populations on external datasets to make it ready for full clinical use and investigated regarding the incorporation of multimodal patient information.