Building Segmentation from High-Resolution Remote Sensing Imagery: A Comparative Evaluation of Convolutional and Transformer Architectures

Abstract

Architectures for building extraction are usually compared under different training and inference set-ups, which makes the reported performance figures incommensurable. This study evaluates three convolutional segmentation architectures (U-Net, U-Net++, DeepLabV3+) and one transformer-based architecture (SegFormer) on the aerial orthophoto subset of the WHU Building Dataset under a single training and inference protocol. Fixing the encoder to ResNet-34 in the convolutional models separates the effect of decoder design from backbone capacity. U-Net++ achieved the highest performance at 89.04% IoU and 94.20% F1, and the difference between the models remained within a narrow range of 2.42 points on the standard IoU metric. On the boundary IoU metric, which measures overlap only within the edge band, the same spread rose to 9.26 points, roughly 3.8 times as wide. Reporting standard IoU alone therefore understates the difference between architectures systematically. An inference protocol combining an exponential moving average, eight-way test-time augmentation and a decision threshold tuned on the validation set yielded gains of 0.47 to 0.69 IoU points without changing the ranking of the models; that magnitude corresponds to roughly a quarter of the total spread attributable to architecture. On the cost side, parameter count proved a misleading indicator: a 6.7% difference in parameters between U-Net++ and U-Net corresponded to a 135.2% difference in multiply-accumulate operations. The results show that model selection in building extraction cannot rest on a single accuracy figure, and that boundary-based metrics should govern the choice in applications where the position of the boundary is critical.