GA-Gabor guided swin-UNet for retinal vessel segmentation

Abstract

Retinal vessel segmentation is a significant technique for assisting clinical diagnosis of fundus diseases. Although deep learning-based approaches have achieved significant progress in recent years, existing models still exhibit limitations in modeling high-frequency features of thin vessel structures, particularly in catching direction-aware features and representing information across multiple scales. To address this issue, this study proposes a GA-Gabor-guided Swin-UNet for retinal vessel segmentation. Specifically, a genetic algorithm is first employed to optimize the parameters of a multi-orientation and multi-scale Gabor filter bank in an offline manner, so that the resulting filters can better match retinal vascular structural characteristics under different data distributions. The optimized Gabor responses are then embedded into the Swin-UNet encoder as a vessel-aware structural prior. Furthermore, a gated residual modulation mechanism is introduced to adaptively control the contribution of the enhanced responses, enabling the network to strengthen informative vascular features while suppressing noise interference. In this way, the proposed framework combines the direction-sensitive local enhancement ability of Gabor filters with the global dependency modeling capability of Swin-UNet. Experiments on three public retinal vessel datasets, namely DRIVE, CHASEDB1, and HRF, demonstrate the effectiveness of the proposed method. On the DRIVE dataset, the proposed approach yields an Accuracy of 0.9780, a Sensitivity of 0.8509, and an AUC of 0.9910. For the CHASEDB1 dataset, the corresponding values are 0.9849, 0.8937, and 0.9950, respectively. On the HRF dataset, the model achieves 0.9783 in Accuracy, 0.8324 in Sensitivity, and 0.9876 in AUC. The experimental results demonstrate that the proposed GA-Gabor module effectively enhances direction-sensitive feature representation and improves the model’s capability to identify fine vascular structures. The findings highlight the potential of the proposed method to advance computer-aided diagnostic systems for retinal diseases.