X-Net: Hybrid Transformer–CNN Framework with Multi-Aspect Attention for Precise Colon Polyp Segmentation

Abstract

Accurate localization of colon polyps plays a crucial role in the early diagnosis of colon cancer. In this paper, we propose a novel network architecture, X-Net, which integrates the Pyramid Vision Transformer (PVT) and EfficientNet-B0. We selected EfficientNet-B0 due to its optimized design and computational efficiency, enabling it to effectively capture fine-grained local features. Meanwhile, we use PVT V2-b1 (Pb1) because of its hierarchical structure and self-attention mechanism, which enhance the model’s ability to preserve multi-scale features. By concatenating these two networks in part of the decoder, the proposed model effectively leverages both local and global features. Additionally, the X-Net architecture introduces three blocks: Shape-Aware Enhancement Block (SAEB), Multi-Scale Feature Block (MSFB), and Multi-Aspect Attention Block (MAAB). The SAEB utilizes differences in initial-layer features extracted from the encoder to filter noise in polyp images. The MSFB extracts and enhances multi-scale features from the backbone, effectively integrating semantic information, which improves polyp segmentation accuracy and enhances robustness against variations in polyp size and shape. Finally, the MAAB employs high-level features from the outputs of the MSFB and SAEB to attend to the foreground, background, and boundaries. To address the class imbalance problem, we propose a hybrid loss function combining Tversky Loss and Binary Cross-Entropy Loss. For evaluation, we trained the proposed network on four polyp datasets: Kvasir-SEG, CVC-ClinicDB, CVC-T, and ETIS. The results demonstrate that the model performs well across different datasets with varying image quality and characteristics.