Hierarchical fine-grained alignment and dynamic gated fusion for Mamba-based referring remote sensing image segmentation

Abstract

Referring Remote Sensing Image Segmentation (RRSIS) addresses the task of identifying ground objects guided by natural language descriptions. While Transformer-based architectures have shown promise, significant limitations remain in processing high-resolution imagery due to quadratic computational complexity and the lack of fine-grained multimodal alignment across diverse object scales. This paper proposes a novel framework HFA-Mamba, centered on the MambaVision architecture to exploit global receptive fields with linear complexity. Specifically, we introduce a Fine-Grained Alignment with Hierarchical Injection (FGA-HI) module, which employs a sequence-concatenation strategy for semantic decoupling. FGA-HI routes positional cues to shallow encoder stages and entity-scale attributes to deeper stages, ensuring precise hierarchical semantic guidance. Additionally, to manage extreme scale variations, we develop a Text-Conditioned Dynamic Multi-Scale Gated Fusion (TCD-MSGF) module. By integrating large-kernel depthwise separable convolutions with a text-driven residual gating mechanism, TCD-MSGF adaptively refines spatial receptive fields without degrading pre-trained visual features. Quantitative evaluations on the RefSegRS dataset reveal that the proposed method achieves a mIoU of 69.35%, significantly surpassing established benchmarks such as LGCE. Ablation analysis validates the efficacy of the hierarchical injection and the synergistic effect of the dynamic gating mechanism in enhancing multimodal reasoning.