Early Detection and Subtype Classification of Lung and Colon Carcinomas Using an Optimized Hybrid Deep Neural Network

Abstract

The accurate and early identification of carcinoma subtypes in histopathological images remains a critical challenge in reducing mortality rates associated with lung and colorectal cancers. This paper presents a novel optimized hybrid deep neural network (OHDNN) framework specifically designed for early detection and subtype-level classification of pulmonary and colorectal carcinomas. The proposed architecture synergistically integrates a modified DenseNet201 backbone for hierarchical feature extraction with a cross-attention transformer module for capturing long-range spatial dependencies across histopathological tissue architectures. A multi-objective optimization strategy combining Grey Wolf Optimizer (GWO) and Cuckoo Search (CS) algorithms is introduced for adaptive hyperparameter tuning, including learning rate scheduling, dropout regularization, and layer-wise depth selection. The model is trained and rigorously evaluated on the publicly available LC25000 dataset, comprising 25,000 high-resolution histopathological images across five carcinoma subtypes. Experimental results demonstrate that the proposed OHDNN achieves superior performance with 98.9% accuracy, 98.7% precision, 98.6% recall, and 98.6% F1-score, significantly outperforming conventional CNN architectures, standalone vision transformers, and existing hybrid approaches. The integration of dual optimization mechanisms reduces overfitting by 34% compared to non-optimized baselines while improving convergence speed by 28%. Gradient-weighted class activation mapping (Grad-CAM) analysis confirms that the model attends to pathologically relevant tissue regions, enhancing clinical interpretability. This work establishes the OHDNN as a scalable, interpretable, and highly accurate solution for AI-assisted early carcinoma diagnosis, with significant implications for precision oncology and digital pathology workflows.