From Relation to Structure: Spatial-Semantic Guidance and Structure Refinement Network for Multimodal Remote Sensing Segmentation

Abstract

Semantic segmentation is a fundamental task in remote sensing image (RSI) interpretation, aiming at pixel-wise classification of land cover. Recently, multimodal RSI semantic segmentation has attracted significant attention for its ability to alleviate the information bottleneck inherent in unimodal methods. However, existing multimodal methods often emphasize feature alignment and complementarity while overlooking the inherent coexistence and mutually exclusive relationships among land covers, frequently resulting in fragmented predictions. Moreover, achieving fine-grained boundary segmentation remains a crucial challenge. To address these challenges, we propose R2SNet, a spatial-semantic guidance and structure refinement network for multimodal RSI segmentation. The core innovation of R2SNet lies in a progressive optimization strategy that guides from semantic relations modeling to spatial structure refinement via three collaborative modules. Specifically, the asymmetric dual-branch (ADB) encoder performs modality-specific extraction of semantic and geometric features to mitigate computational redundancy and modal interference. Subsequently, the spatial-semantic guided cross-modal fusion (SGCF) module explicitly embeds spatial-semantic priors into the fusion process to suppress unreasonable relationships. Finally, the progressive structure refinement (PSR) decoder aggregates multiscale contexts and employs an eight-directional selective scanning mechanism to enhance structural integrity and refine boundaries. Extensive experiments on the ISPRS Vaihingen, ISPRS Potsdam, DDHR, and WHU-OPT synthetic aperture radar (SAR) datasets demonstrate that R2SNet outperforms existing state-of-the-art methods, particularly regarding semantic consistency and boundary precision. The code will be available at: https://github.com/bnu-wgy/R2SNet