Skip to content
Book Open access

Full-tuning CLIP for Weakly Supervised Semantic Segmentation

Nov 2026 · 0 citations · 49 references

Abstract

Weakly supervised semantic segmentation (WSSS) with image-level labels has successfully bridged the gap between weak annotations and pixel-wise predictions based on CAMs. Recently, Contrastive Language-Image Pre-training (CLIP) has been introduced into WSSS to improve the quality of CAMs and yields promising results. Existing methods predominantly rely on a frozen CLIP and repose its pre-trained knowledge can seamlessly generalize to the downstream segmentation datasets. However, such frozen representations are doomed to lack the vital adaptability to overcome inherent distribution shifts between the CLIP's training set and the segmentation datasets without task-specific fine-tuning. In this manuscript, we propose a Self-Supervised Distribution Alignment (SSDA) strategy to address this issue. Specifically, SSDA leverages the pseudo-labels generated by CLIP Surgery to reconstruct the segmentation dataset into an image-text matching dataset for full-tuning CLIP. During the training process, SSDA introduces a Feature Rectification Loss L_FR to supervise the CAMs of the vanilla CLIP by the CAMs of CLIP Surgery, thereby achieving the goal of forcing the patch tokens to attend to the object itself rather than the frequently co-occurring background regions. Furthermore, we propose a SAM-based post-processing (SPP) strategy to generate better pseudo labels by refining the CAMs produced by the CLIP. SPP utilizes SAM-generated attention as a spatial guidance to suppress interference from irrelevant regions during affinity propagation, while simultaneously employing refined pseudo labels as prompts for SAM to yield more fine-grained masks. Experiments show that our method achieves the state-of-the-art performance on the PASCAL VOC 2012 dataset and competitive results on the MS COCO 2014 dataset.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.