A Hybrid Deep Learning Model for Colorectal Polyp Detection and Classification From Endoscopic Images

Abstract

Globally, colorectal cancer (CRC) remains a key contributor to cancer‐related death, with most malignancies developing through the progression of colorectal polyps. Early detection and accurate histological classification of polyps during colonoscopy are essential for effective CRC screening and prevention. However, conventional colonoscopy may fail to detect certain lesions and exhibits variability in diagnostic performance due to operator dependence and challenging imaging conditions. To address these limitations, this study proposes a hybrid deep learning (DL) model that integrates YOLOv10 for polyp detection and feature extraction with a customized Convolutional Neural Network (CNN) for the histological classification of colorectal polyps into hyperplastic and adenomatous categories. A total of 6000 endoscopic images obtained from the Harvard Dataverse PolypsSet repository were used for model development and evaluation. To improve robustness and generalization, data augmentation techniques were applied during training, and stratified 5‐fold cross‐validation was employed to prevent data leakage between training and validation sets. Experimental results demonstrated that the proposed YOLOv10–CNN model achieved an average detection mAP@50 of 0.9848 and a classification accuracy of 0.9913 across the cross‐validation folds. External validation on an independent dataset achieved mAP@50 of 0.926, indicating good generalization ability to unseen data. Furthermore, the model achieved an inference speed of approximately 120 frames per second (FPS), demonstrating efficient computational performance. A web‐based graphical user interface was also developed to facilitate visualization of detection and classification results from colonoscopy videos. The findings suggest that the proposed hybrid model provides accurate and efficient polyp detection and classification while maintaining stable performance across internal and external evaluations. The proposed approach may serve as a supporting computer‐aided analysis tool for colorectal polyp screening.