HDSMNet: Height-Guided Sparse Cross-Modal Fusion for High-Resolution Remote Sensing Semantic Segmentation

Abstract

High-resolution remote sensing semantic segmentation requires the joint modeling of local details, global semantics, and height-derived geometric structures, and it provides an important basis for urban object mapping, land-cover analysis, and fine-grained spatial understanding. However, in complex urban scenes, fine-grained boundaries, small objects, inter-class similarity, and spectral confusion can still weaken the stability of pixel-level prediction. To enhance discriminative dense feature representations in high-resolution remote sensing images, we propose HDSMNet, a dual-branch multimodal semantic segmentation network designed for optical–nDSM data. The network separately extracts appearance and semantic features from optical imagery and height–structural features from nDSM, and introduces a Height-Guided Sparse Cross-Modal Fusion (HGSCF) module. Rather than treating nDSM as an additional feature source for generic fusion, HGSCF derives contextual representations, local feature contrasts, and structural-discontinuity cues from encoded nDSM features and uses them to guide sparse anchor-based interaction between optical and height features. This design enhances discriminative dense feature representations through interaction with a compact set of geometry-guided anchors. To complement HGSCF at the output stage, HDSMNet further adapts a Context-Guided Refinement (CGR) path that combines intermediate-response-guided contextual aggregation with dynamic feature modulation. This supplementary path recalibrates decoder features for output refinement. Experiments on the ISPRS Potsdam and Vaihingen datasets show that HDSMNet achieves mIoU values of 86.57% and 84.22%, respectively; ablation results further identify HGSCF as the main contributor to the observed improvement.