RLL-YOLO-seg: an improved instance segmentation model for pixel-level steel surface defect detection

Abstract

Steel surface defect detection is important for ensuring product quality and production safety. To address the limited region-level description provided by conventional localization methods, this paper proposes RLL-YOLO-seg, an improved instance-segmentation model built on YOLOv11n-seg (hereafter, the baseline). The C3K2-RVB module improves the representation of small defects and local textures through multi-scale convolution, residual learning, and squeeze-and-excitation. The SPPF-LSKA module combines spatial pyramid pooling with large separable kernel attention to improve multi-scale fusion and boundary-detail capture. Segment-LQE decouples localization from mask generation and estimates localization quality to improve segmentation accuracy and boundary integrity. On NEU-Seg, RLL-YOLO-seg achieves Box mAP@0.5 and Mask mAP@0.5 of 83.6% and 76.9%, respectively, improving on the baseline by 5.2 and 4.1 percentage points. On GC7, it achieves 84.0% Box mAP@0.5 and 80.0% Mask mAP@0.5, improvements of 0.7 and 3.8 percentage points. These results demonstrate accurate localization and segmentation of steel-surface defect instances.