Spatial-Aware Consistent Panoptic Gaussian SLAM for Open-Set Scene Understanding

Abstract

Panoptic and geometric scene understanding are essential for robots operating in complex environments. While recent advances in 3D Gaussian Splatting (3DGS) provide a strong geometric foundation for SLAM, integrating open-set 2D panoptic predictions into a consistent long-term 3D map remains challenging. Two common inconsistency problems arise from pure 2D supervision: globally, the lack of a 3D constraint in label assignment causes the system to erroneously assign the same identity to different objects that never co-appear in the same 2D frame. Locally, features supervised exclusively by 2D predictions lack spatial regularization, resulting in noisy 3D segmentations. We propose a panoptic Gaussian SLAM framework that addresses both failure modes while remaining efficient and monocular RGB-only. For global instance association, we augment standard 2D matching with lightweight 3D constraints derived from the continuously updated Gaussian map. As the Gaussian map is continuously updated, earlier assignments can be revised, reducing identity collisions and sensitivity to depth-scale deviations.For local coherence, Laplacian smoothing over neighboring Gaussians regularizes semantic and instance features, yielding cleaner 3D segmentations. Together, these components form an online system that produces consistent panoptic segmentations and clean object-level 3D assets.