Differential morphological profile neural networks for segmentation of remote sensing imagery

Abstract

Semantic segmentation, assigning a class label to each pixel, has been revolutionized by deep neural networks. A significant milestone, the Fully Convolutional Network (FCN), demonstrated that a purely convolutional architecture could outperform previous approaches. Subsequent architectures largely adopted its encoder–decoder pattern. Models like U-Net and the DeepLab series advanced convolutional segmentation, and transformer-based designs such as SegFormer and EfficientViT achieved even higher accuracy. However, remote sensing imagery, often captured from an overhead view, differs significantly from the social media-like datasets on which these models are typically benchmarked. Remote sensing imagery also introduces the challenges of multi-scale objects, varying resolutions, complex scenes, extreme background/foreground imbalance, large intra-class variation, and small inter-class variation. Specialized architectures (FarSeg, AerialFormer) and benchmarks (iSAID) exist, but remote sensing remains challenging for segmentation. One approach for improving performance in remote sensing imagery is the use of the Differential Morphological Profile (DMP). Initially developed for satellite image segmentation, the DMP encodes explicit shape information for a given input image via the application of grayscale morphological operations. It has been incorporated into deep neural networks called DMPNets which take as input this shape information and produce features for tasks such as classification and object detection of overhead remote sensing imagery. This dissertation extends DMPNet to semantic segmentation of remote sensing imagery. A Direct-In integration strategy, in which the input image is converted to grayscale and combined with its morphological profile, consistently underperforms RGB-only baselines across six architectures and three benchmark datasets, with an ablation study confirming that the loss of color information is the primary cause. Applying the Hybrid DMP dual-stream design from prior DMPNet work, which processes RGB and DMP inputs through parallel encoders before feature fusion, substantially recovers this deficit and in many configurations surpasses the RGB baseline, with the largest gain of +3.19 mIoU observed for SegNeXt-S on the ISPRS Potsdam dataset. Several alternative fusion strategies beyond the original concatenation method are evaluated, including element-wise operations and squeeze-excitationbased attention, with squeeze-excitation methods emerging as the most consistently competitive across architectures and datasets. Per-class analyses reveal that object classes with strong geometric regularity, such as roundabouts, cars, and agricultural fields, benefit most from DMP integration, while classes whose primary discriminative cue is spectral appearance, such as swimming pools and barren land, are most harmed by the loss of color information under the Direct-In strategy. Experiments with remote sensing-specific architectures show that these specialized designs already capture much of the shape information the DMP provides, yielding more modest Hybrid DMP gains than for general-purpose architectures. Notably, general-purpose architectures enhanced with Hybrid DMP are competitive with, and in some cases exceed, the purpose-built remote sensing architectures. An extension to instance segmentation via Mask-RCNN reveals that the DMP provides complementary but not dominant signal in detect-then-segment pipelines, where spatial cropping attenuates the globally coherent morphological features. Together, these findings establish that DMP features offer useful complementary shape information for remote sensing segmentation, but that preserving RGB information through a dual-stream design is critical to realizing this benefit.