MSSAM: Multiscale Spatial-Aware Mamba for Multimodal Remote Sensing Image Classification

Abstract

Applying deep learning techniques to classify multimodal remote sensing imagery plays a vital role in Earth observation. Current methods primarily face three challenges: first, interference from complex backgrounds, the presence of strongly directional structures such as roads and building clusters, and significant variations in the scale of ground objects. Second, conversely, traditional transformers are capable of modeling global dependencies, but the quadratic complexity of self-attention limits their applicability to high-resolution remote sensing data. Finally, emerging state-space models, such as Mamba, offer the advantage of linear computational complexity. However, their unidirectional scanning mechanism creates a structural contradiction with 2-D spatial understanding, leading to directional bias and contextual fragmentation. To overcome these limitations, we propose multiscale spatial-aware Mamba (MSSAM), a groundbreaking deep learning architecture that integrates the multiscale spatial-aware block, which employs a bidirectional context pre-fusion mechanism enabling a single Mamba module to capture bidirectional dependencies, and introduces dual-view Transformers to explicitly model spatial dependencies along horizontal and vertical dimensions respectively, achieving comprehensive 2-D context awareness, and the multibranch attention spectral rearrange block, which introduces a novel spectral rearrange mechanism that systematically promotes cross-branch feature interaction by reorganizing spectral features across channel groups. This enables the decoder to more effectively exploit local details and multiscale semantic information through cascaded channel and spatial attention-guided multiscale depthwise convolutions. On the Muufl, Houston, and Augsburg datasets, MSSAM achieves classification accuracies of 96.57%, 99.79%, and 97.89%, respectively, substantially outperforming existing state-of-the-art methods in comprehensive evaluations. Furthermore, MSSAM exhibits superior generalization performance when tested on two single-modality datasets: Pavia University (hyperspectral) and Flevoland (PolSAR). This research introduces a novel framework for intelligent interpretation of multimodal remote sensing imagery.