A modular mapping architecture is demonstrated that establishes 3D Semantic Scene Graphs (3DSSGs) as its foundational back-end, enabling the dense representation of extensive environments containing thousands of unique object instances and supporting open-vocabulary queries via CLIP features without requiring any additional post-processing steps.
3D Gaussian splatting (3DGS) has recently emerged as an efficient and scalable method for high-fidelity 3D scene reconstruction, representation, and real-time rendering. In addition to geometric reconstruction, increasing research attention focuses on enriching 3D Gaussian primitives with semantic information, which can be related to an arbitrary application or domain, as well as common knowledge. However, the existing approaches to semantic 3DGS significantly differ in how semantics are represented, learned, and accessed, which makes systematic analysis difficult. This paper provides a review on semantic extensions to 3DGS. We introduce a unified multi-axis taxonomy that enables us to classify the available methods in terms of five complementary categories: semantic vocabulary space, representation form, functional role, knowledge source, and query mechanism. The analysis reveals key design trade-offs related to the flexibility, efficiency, and semantic expressiveness of the methods. Furthermore, we review datasets, benchmarks, and evaluation metrics used in the field, indicating the diversity of approaches and the lack of common evaluation frameworks. Based on this analysis, we also identify open challenges and possible future research directions. The presented survey is relevant to advances in games and immersive technologies, where semantically enriched real-time 3D representations are essential for interactive environments, AR/VR, and intelligent scene understanding. The systematic analysis presented in this survey aims to facilitate a deeper understanding of semantic 3DGS and support the development of more general, efficient, and task-aware 3D scene understanding systems.
Indoor 3D Scene Graphs (3DSGs) represent environments as multi-layer hierarchies that connect observed geometric primitives (e.g., planes) to higher-level metric-semantic concepts (e.g., rooms, floors, buildings), enabling incremental spatial reasoning for robotic perception and SLAM. However, classical high-level concept generation approaches rely on hand-crafted rules for specific concept classes, while learning-based methods require separate models for graph structure and spatial node features (e.g., centroids), which limits scalability to novel classes and more complex hierarchies. We propose a unified autoregressive diffusion-based graph generative model that jointly learns structure and features, constructing complete 3DSGs bottom-up from observed vertical planes across arbitrary hierarchy depths. Our method consistently surpasses all learning-based and random baselines across 3DSG datasets spanning synthetic scenes, real architectural floor plans, and robotic sensor data, with varying layout complexity and hierarchy depth, and surpasses a one-shot model with oracle access to the target graph size on the largest hierarchy and on real single-floor data. Finally, we propose an adaptation of the Fused Gromov--Wasserstein distance for principled graph-level evaluation of generated 3DSGs against ground truth.
J. A. Millan-Romera, Samuel Cognolato, Holger Voos et al.· 0 citations
Open vocabulary 3D scene understanding is essential for next-generation interactive systems, empowering users to intuitively query and navigate reconstructed environments using natural language. However, current 3D Gaussian frameworks are often bottlenecked by restrictive multiview capture requirements, costly scene-specific optimization, and the massive memory overhead of storing dense language features. We present a novel, training-free pipeline that fundamentally reimagines this paradigm by explicitly decoupling 3D geometric reconstruction from semantic integration. Given a standard monocular video sequence as input, our method efficiently outputs a compact, highly interpretable, and fully searchable object-level semantic Gaussian map. Rather than entangling heavy language embeddings within the mapping loop, we extract geometry independently and ground semantics through a lightweight, modular post-processing framework. Extensive evaluations on the Replica dataset demonstrate that this decoupled architecture preserves strong rendering fidelity and competitive segmentation accuracy. Crucially, by replacing dense per-Gaussian storage with modular, object-level semantic embeddings, our approach delivers an order-of-magnitude reduction in memory usage compared to SOTA baselines. This provides a highly efficient, scalable, and practical solution for open-vocabulary 3D retrieval and question answering directly from everyday monocular video.
Pouya Ardekhani, Zahra Dehghanian, Morteza Abolghasemi et al.· 0 citations
Abstract. We demonstrate an end-to-end pipeline for 3D scene understanding which integrates unsupervised graph-based point cloud segmentation with LLM-enabled spatial reasoning and editing. A point cloud is segmented into a SemanticPatch decomposition (stage 1), labeled using a zero-shot vision-language model (stage 2; SAMv2, CLIP), encoded into a scene graph in the latent space (stage 3) capturing geometry, topology, and constraints, and finally manipulated by an LLM-based agent (stage 4) to execute a specified editing task. The LLM agent can be instructed by natural language input to reason about a scene graph and a point cloud, compute a geometric transformation for the input point cloud, and check its own output against a set of constraints (e.g. ADA-compliance). We validate our approach on three different point clouds: a classroom (Leica RTC360, 1.3 M points), a construction site (NavVis VLX mobile scanner, 4.4M points), and the Paris-Lille-3D benchmark. Our segmentation approach scores 97–99% on the fitness score and 92–99% on the F1-score across all three benchmarks. Our LLM agent solves reconfiguration tasks in 1–10 min, achieving a 100% constraint-satisfaction rate and outperforming a human annotator.
Florent Poux, Alex Key· The International Archives o...· 0 citations
3D Semantic Scene Completion (SSC), a cornerstone task in computer vision, aims to simultaneously predict the geometric structure and semantic content of 3D scenes from sparse inputs. This capability is crucial for advancing applications in robotics, autonomous driving, and virtual reality. However, SSC faces significant challenges, including the high computational cost of capturing long-range contextual information and the scarcity of 3D semantic labels leading to overfitting. To address these limitations, we propose an enhanced network for semantic scene completion. Firstly, we devise a 3D Local- Global Linear Attention Mechanism (LG-LAM) that efficiently captures long-range contextual information with linear complexity, enabling a comprehensive understanding of the 3D scene without heavy computational burdens. Secondly, a 3D Feature Augmentation Module (FAM) is integrated to enrich feature diversity through rotation-invariant learning, mitigating overfitting and enhancing the model’s robustness given limited annotations. Extensive experiments on the NYUCAD dataset demonstrate that our method achieves state-of-the-art performance among non-iterative methods while introducing negligible computational overhead.
Jie Li, Jiaheng Xu, Laiyan Ding et al.· International Conference on...· 0 citations
We study open-vocabulary 3D indoor layout generation, which synthesizes diverse and physically plausible scenes from unlabeled 3D assets using free-form language instructions. Recent methods leverage large language models (LLMs) and vision-language models (VLMs) to generate structured scenes from text. However, most model inter-asset relations implicitly or rely on local pairwise constraints and local optimization. These formulations are poorly aligned with the global, highly non-convex layout space, often yielding locally plausible yet globally inconsistent or physically infeasible scenes. We address this problem with a graph-based intermediate representation that separates semantic coherence from physical feasibility, together with a hybrid search-and-refinement strategy. First, Global Semantic Verification (GSV) represents scenes as structured graphs and enforces semantic constraints through rule-based verification. This explicit validation removes contradictory configurations and produces a globally consistent semantic scaffold. Second, Global Physical Feasibility Search (GPFS) combines evolutionary search for global exploration with gradient-based refinement for local exploitation. It reduces dependence on VLM-proposed initialization and improves robustness in non-convex and discontinuous feasible spaces. Together, GSV and GPFS move layout generation beyond local relational modeling and initialization-sensitive optimization toward globally consistent reasoning and search. Experiments show that our method achieves state-of-the-art performance in open-vocabulary 3D indoor layout generation, improving both semantic consistency and physical plausibility.
Jialu Huang, Yingxuan You, Fei Wang et al.· 0 citations