Multimodal LLM for Function-Aware Building Polygon Extraction From Remote Sensing Imagery

Abstract

Building function mapping from remote sensing (RS) imagery is essential for fine-grained urban understanding. However, pure vision-based building function prediction remains highly challenging due to the complex spatial and semantic relationships among urban geospatial entities. In this letter, we propose a multimodal large language model (LLM)-based method for building function polygon extraction, termed multimodal LLM-based building function polygon extraction model (MLBFM). First, MLBFM extends segmentation tokens in the LLM vocabulary to a multiclass version for encoding category-wise function information and introduces external polygon and function decoders for attributed polygon prediction. Second, to further enhance the embedding-as-mask paradigm, the internal embeddings of the multimodal LLM corresponding to the multiclass segmentation tokens are decoupled into spatial and semantic components, which are, respectively, utilized for position-guided feature encoding and semantically constrained function classification. Experiments on the publicly available BuildingSense and building footprint function (BUFF) datasets demonstrate that MLBFM achieves competitive performance in both building function classification and joint building function classification with polygon shape delineation. The implementation of MLBFM will be available at https://github.com/yudawen/MLBFM