Identifying Urban Building Functions via Vision–Language Alignment of Remote-Sensing Imagery and POI Semantics

Abstract

Urban building function identification plays a fundamental role in fine-grained urban sensing, data-driven spatial planning, and sustainable urban management. Existing approaches either rely on handcrafted features derived from a single data source or adopt shallow feature integration strategies, thereby limiting deep cross-modal interactions between remote-sensing visual features and sociofunctional semantics derived from point-of-interest (POI) data. With the rapid development of multimodal pretrained models, contrastive-learning-based approaches enable visual and textual representations to be aligned within a shared semantic space, facilitating high-level cross-modal representation learning. Motivated by this advancement, this study presents a vision–language representation learning framework for urban building function identification. Specifically, first, a multimodal dataset, building understanding with integrated local textures and POI in the Yangtze River Delta, was constructed by linking building-level remote-sensing images with surrounding POIs and organizing their spatial and semantic information to facilitate unified multimodal representation learning; second, a multimodal fusion architecture integrating the sigmoid loss for language image pretraining and the segment anything model was proposed, which fuses POI-derived sociofunctional semantics with remote-sensing visual features, and further incorporates mask-guided cross-modal attention reasoning to achieve fine-grained building function classification; and third, ablation studies and visual analyses were conducted to evaluate the effectiveness of the proposed method quantitatively and qualitatively. This study presents a promising vision–language representation learning framework for fine-grained urban building function identification and offers insights for developing geographically context-aware urban intelligence models.