Progressive Multi-Layer Decoupling for Spatially-Enhanced Open-Vocabulary Perception

Abstract

Scaling up vision-language models like CLIP for dense prediction tasks has recently gained considerable attention. DeCLIP builds upon this direction by decoupling the self-attention mechanism in the final transformer layer to separate content and context representations. However, evidence suggests that different layers in a vision transformer encode qualitatively different visual information, raising the question of limiting decoupling to only the last layer is truly optimal. In this work, we extend DeCLIP by decoupling the self-attention modules in the last six layers of a CLIP ViT. Our experiments on COCO-Panoptic reveal a notable performance shift: stuff classes improve by +3%, while thing classes degrade by 2%. Cross-dataset evaluation on ADE20k confirms the same trend, with consistent gains on stuff classes such as water and sky. These results expose an inherent stuff-thing trade-off in layer-wise feature decoupling, suggesting that deeper semantic layers naturally favor amorphous background categories over instance-level objects. Our work provides new insights into the role of hierarchical feature decomposition for open-vocabulary dense perception, and highlights the need for class-aware decoupling strategies in future research.