Jul 2026· Automation in Construction· Vol 190, pp. 107117· 0 citations· 57 references
Computer Science
TL;DR
Building-MLLM is proposed, a point cloud-centered multimodal large language model (MLLM) for indoor components, which models point clouds and instructions to generate responses across Simple Recognition, Complex Captioning, and Multi-Engineering Question Answering tasks, demonstrating superior indoor component language understanding and providing initial generalizability in transfer inference on other real-world datasets.
Abstract
Point cloud-based understanding has become an important enabler for facility operation and maintenance involving indoor building components. However, existing methods output only discrete labels without explaining component functions or natural language interactions. This paper proposes Building-MLLM, a point cloud-centered multimodal large language model (MLLM) for indoor components, which models point clouds and instructions to generate responses across Simple Recognition, Complex Captioning, and Multi-Engineering Question Answering tasks. Building-MLLM addresses semantic concentration through four domain-specific mechanisms: Point Information Enhancer for task-relevant semantics, Geometry-Preserving Regularization preventing geometric erosion, fixed textual prefix for domain stabilization, and multi-dimensional LoRA balancing recognition with reasoning. A multi-constraint progressive instruction-generation engine is developed to compile a synthetic point cloud-text dataset with 4198 objects, 37,782 instruction-following pairs, and 47 categories. Experiments show that Building-MLLM achieves 88.00%, 65.10%, and 68.14% on the three task types, respectively, demonstrating superior indoor component language understanding and providing initial generalizability in transfer inference on other real-world datasets.
This work introduces SPATIALQUERY, a training- free framework for CIDQ reasoning from a single RGB image, together with SPATIALQUERY-1M, a benchmark containing over one million RGB-only question-answer pairs from 200 indoor scenes, and proposes Uncertainty-Aware Chain-of-Thought (UA-CoT) prompting, which incorporates geometry- derived per-instance uncertainty into the VLM reasoning process.
The development of 3D large multimodal models (3D-LMMs) has advanced research on 3D visual question answering. Yet most existing methods rely on implicit feature mapping, where point clouds or scene features are directly projected into the latent space of a language model, without explicitly modeling local spatial structure. In complex indoor environments, this design can lead to spatial judgments that are inconsistent with the actual physical layout during 3D visual question answering, a phenomenon referred to as spatial hallucination. To address this issue, this paper proposes TA-LMM, a 3D visual question answering method built on explicit scene graphs and local topological priors. The method begins by parsing a raw 3D scene into an explicit spatial-semantic scene graph and extracting instance-level representations that encode both semantic features and geometric location information. It then constructs a local physical neighborhood around the target object, serializes neighboring objects together with their distance information into structured priors, and injects them into the multimodal reasoning sequence as conditional context. Under the current experimental setting on the Replica dataset, the results show that the proposed method achieves strong performance on spatial-relation question answering while also alleviating spatial hallucination to a certain extent. These findings suggest that explicit local topological priors can improve scene consistency in 3D visual question answering.
Kaixin Wu, Kunlin Zhou, Boxin Li et al.· International Conference on...· 0 citations
GSP++ is presented, a graph-based inference framework that exploits the manifold structure of test-time point clouds to refine VLM scores without additional training, and introduces a self-training strategy that selects high-confidence positive and negative samples and assigns them calibrated pseudo scores to further stabilize propagation.
Tiankai Chen, Yushu Li, Adam Goodge et al.· IEEE Transactions on Pattern...· 0 citations
Semantic 3D city models provide rich geometric and semantic information, but remain challenging for non-experts and interdisciplinary researchers to access and query due to their complex structures and specialized data formats. To address this issue, we present CityLLM, a framework for natural-language querying of semantic 3D city models alongside complementary urban datasets. The framework combines spatial and graph databases within an LLM-based workflow that supports iterative query refinement and cross-database chaining. We evaluate CityLLM on a CityJSON dataset of Rotterdam (853 LoD2 buildings) using GPT-OSS, Gemini 3.1, and GPT-5.4, along with selected variants, across multiple metrics: answer correctness, visualization correctness, query success, and retry attempts. A total of 54 natural-language queries are curated across four scenarios: spatial, graph, cross-database, and conversational. Results show strong overall performance, with answer correctness ranging from 85.2% to 100%, visualization correctness from 92.9% to 100%, a 100% query success rate, and fewer than three retries across all 54 queries. Overall, the findings suggest that CityLLM provides a lightweight and extensible approach for conversational access to semantic 3D city data.
Rabindra Lamsal, S. Zlatanova, John Shen· 0 citations
Instance-Enriched Semantic Maps is proposed, a unified framework with three key contributions: instance-level two-and-a-half-dimensional rich information mapping, storage-efficient semantic representation that achieves approximately 96% reduction compared to three-dimensional scene-graph approaches while preserving sufficient spatial information for navigation, and robust query processing via LLM-based target selection.
J. Hong, Eunae Kang, Sanghyun Kim et al.· Engineering applications of...· 0 citations
Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage
Xianhao Chen, Jiarui Hu, Yuanbo Yang et al.· 0 citations