Jun 2026· arXiv.org· Vol abs/2606.30638· 0 citations· 45 references
Computer Science
TL;DR
GaussDet is presented, a method that circumvents the need for dense CLIP features by leveraging discrete, open-vocabulary 2D object detectors with referring expression capabilities and achieves a substantial 16.7% mIoU improvement in referential grounding within a strict zero-shot setting.
Abstract
3D Gaussian Splatting (3DGS) has emerged at the forefront of 3D scene reconstruction. Extending 3DGS with language-driven, open-vocabulary understanding has gained significant attention for real-world applications such as embodied AI. Recent methods achieve this by learning an instance feature attribute and assigning semantics by distilling high-dimensional Contrastive Language-Image Pretraining (CLIP) features directly into the scene representation. However, the instance grouping mechanisms of these methods either require a predefined number of instances or suffer from noise in their bottom-up grouping strategies. Furthermore, the reliance on CLIP restricts semantic understanding to simple noun phrases, preventing complex spatial reasoning and referential expression grounding. We present GaussDet, a method that circumvents the need for dense CLIP features by leveraging discrete, open-vocabulary 2D object detectors with referring expression capabilities. We learn instance features for individual Gaussians to decompose the scene into 3D instance groups. By rendering these groups and aggregating semantic votes from multi-view 2D detections, we generate a robust View-Aggregated Semantic Label Distribution (VASD) for each 3D instance. This view-aggregation strategy acts as a strong regularizer, attenuating spurious labels caused by low-quality instance grouping. Our approach enables a straightforward, zero-shot extension from simple language queries to complex referential grounding. Extensive evaluations across two key tasks -- open-vocabulary segmentation (LeRF-OVS, ScanNet) and referring expression grounding (Ref-LeRF) -- demonstrate that GaussDet achieves consistent improvements over existing methods. Most notably, we achieve a substantial 16.7% mIoU improvement in referential grounding within a strict zero-shot setting.
ZeroSplat lifts 2D Vision-Language Model priors into 3D space through robust multi-view geometric constraints and enables intrinsic point-level understanding without incurring any additional feature storage, and significantly outperforms state-of-the-art methods across generalized and single-target scenarios while maintaining exceptional efficiency.
Jiayu Ding, Meilu Song, Xiaoyi Zhang et al.· 1 citation
3D Gaussian splatting (3DGS) has recently emerged as an efficient and scalable method for high-fidelity 3D scene reconstruction, representation, and real-time rendering. In addition to geometric reconstruction, increasing research attention focuses on enriching 3D Gaussian primitives with semantic information, which can be related to an arbitrary application or domain, as well as common knowledge. However, the existing approaches to semantic 3DGS significantly differ in how semantics are represented, learned, and accessed, which makes systematic analysis difficult. This paper provides a review on semantic extensions to 3DGS. We introduce a unified multi-axis taxonomy that enables us to classify the available methods in terms of five complementary categories: semantic vocabulary space, representation form, functional role, knowledge source, and query mechanism. The analysis reveals key design trade-offs related to the flexibility, efficiency, and semantic expressiveness of the methods. Furthermore, we review datasets, benchmarks, and evaluation metrics used in the field, indicating the diversity of approaches and the lack of common evaluation frameworks. Based on this analysis, we also identify open challenges and possible future research directions. The presented survey is relevant to advances in games and immersive technologies, where semantically enriched real-time 3D representations are essential for interactive environments, AR/VR, and intelligent scene understanding. The systematic analysis presented in this survey aims to facilitate a deeper understanding of semantic 3DGS and support the development of more general, efficient, and task-aware 3D scene understanding systems.
Camera-reconstructed 3D scenes often require offline visual cleanup before inspection, presentation, or reuse as renderable virtual-scene assets. Representative applications include removing temporary furniture, parked vehicles, equipment, signage, and other distracting or obsolete objects from reconstructed indoor and outdoor environments. Such editing requires not only accurate target localization across viewpoints but also plausible recovery of the previously occluded background. Existing methods often depend on manually specified masks or category-restricted detectors, while projection-based pipelines independently inpaint multiple views and subsequently refine the 3D representation, potentially introducing cross-view appearance and geometry inconsistencies. We present OVR-GS (Open-Vocabulary Removal in Gaussian Splatting), an instruction-driven object-removal framework for pre-trained 3D Gaussian Splatting (3DGS) scenes. Given a free-form instruction, a language parser generates target-oriented queries and a textual background-completion condition. Grounding DINO and the Segment Anything Model (SAM) produce multi-view candidate masks, which are filtered using Contrastive Language–Image Pre-training (CLIP). The proposed Semantic-Aware Gaussian Selector (SAGS) aggregates rendering-contribution-weighted mask evidence, groups spatially coherent candidates, and identifies the target Gaussian subset through rendered-cluster semantic verification. After removal, new Gaussians are initialized from boundary-adjacent primitives and interior samples and optimized locally using Score Distillation Sampling (SDS), while the original background remains fixed. On IMFine, SPIn-NeRF, and Inpaint360GS, OVR-GS achieves peak signal-to-noise ratio (PSNR) values of 19.78, 17.82, and 24.62 dB and Fréchet inception distance (FID) values of 142.30, 148.60, and 34.80, respectively. The results demonstrate the effectiveness of localized Gaussian optimization for instruction-driven cleanup of reconstructed environments before visual inspection, presentation, or reuse as renderable virtual-scene assets.
Yongpeng Ding, Feng Ouyang, Jiawei Fan et al.· Italian National Conference...· 0 citations
This work proposes GroupForward, an instance-grouped feed-forward Gaussian splatting model that reconstructs geometry, appearance, instance structure, and semantics from sparse, unposed, and uncalibrated multi-view images and proposes a Referential Scene Reasoning Framework (RSRF) for complex 3D referring segmentation.
Qijian Tian, Zimeng Wu, Xuhong Wang et al.· 0 citations
OutLangSplat is presented which adapts language Gaussian representations to UAV outdoor scenes by improving feature representation and aggregation reliability, and is the first accessible dataset of open-vocabulary 3D scene understanding for UAV outdoor scenes.
Xiaosheng Yan, Hefeng Wu, Yanghui Xu et al.· 0 citations
This work investigates whether a frozen, self-supervised point transformer already contains the structural information required to isolate object instances without any handcrafted geometric prior, and develops a training-free segmenter that groups points via connected components on a key-similarity graph, using neither density-based clustering nor proximity priors.
Ted Lentsch, Santiago Montiel-Mar'in, Holger Caesar et al.· 0 citations