LS-PolypSeg: A Parameter-Efficient LoRA-Enhanced SAM3 Framework for Polyp Segmentation

Abstract

Accurate polyp segmentation is essential for early detection of colorectal cancer, where precise delineation of lesion boundaries directly impacts clinical decision-making. Despite significant progress, existing convolutional methods often struggle to capture global contextual information, while transformerbased and foundation models introduce high computational complexity and require extensive fine-tuning. In this work, we propose LS-PolypSeg, a parameter-efficient framework that leverages a pretrained SAM3 vision encoder with Low-Rank Adaptation (LoRA) for domain-specific learning. By selectively adapting key transformer layers while keeping most of the encoder frozen, the proposed approach preserves rich pretrained representations while significantly reducing training overhead. To complement global feature extraction, a lightweight UNetstyle decoder performs multi-scale feature fusion, enabling accurate recovery of fine-grained spatial details. Extensive experiments on three benchmark datasets, namely Kvasir-SEG, CVC-ClinicDB, and BKAI-IGH, demonstrate that LS-PolypSeg achieves competitive and state-of-the-art performance across multiple evaluation metrics. These results highlight the effectiveness of combining foundation model representations with efficient adaptation and hierarchical decoding for robust, scalable polyp segmentation.