Semantic-Guided Adaptive Gaussian Segmentation

Abstract

3D Gaussian Splatting (3DGS) enables real-time photorealistic scene reconstruction, yet its segmentation tasks suffer from two critical flaws: poor 3D consistency (e.g., blurred instance boundaries and unstable cross-view semantic association) and insufficient structural awareness near ambiguous object boundaries. To address these issues, this paper proposes a semantic-guided 3D Gaussian segmentation framework. We first process multi-view RGB images and semantic masks to generate 2D semantic codes and initial Gaussian parameters. We then build a Gaussian-level joint representation by combining flow-aligned appearance cues, CLIP-derived semantic descriptors, and explicit Gaussian geometry. Semantic-guided adaptive decomposition identifies boundary-sensitive Gaussians through cross-view boundary statistics and semantic uncertainty, while multi-view voting further refines instance boundaries. Finally, 3D global optimization unifies instance association and labeling. Experiments on ScanNet and SPIn-NeRF show improved results under the reported evaluation protocol and the stated subset definitions. In particular, on the ScanNet benchmark, compared with the evaluated 3DGS baseline InstanceGaussian, SG-AGS improves mAcc@0.25 by 16.2 points for category-agnostic 3D instance segmentation and by 25.6 points for text-query-based open-vocabulary labeling of segmented 3D instances. These results support the usefulness of SG-AGS on the two evaluated benchmarks and suggest potential value for object-level scene querying and digital-twin inspection, while broader generalization to external captures and other scene domains still requires further validation.