An open-vocabulary semantic mapping pipeline is presented that integrates TALOS (TAgging–LOcation–Segmentation–Segmentation) with the probabilistic, instance-aware Voxeland framework and shows a better balance between map completeness, geometric clarity, semantic coherence, and instance separation.
Abstract
High-level robotic tasks, such as those involving planning and interaction, demand a certain degree of scene understanding through suitable representations of the environment that enrich geometric information with object-level semantics, commonly referred to as semantic maps. Traditional techniques to build these maps are restricted to working with a set of categories that is fixed during training, which limits the information the resulting maps can contain. To overcome this limitation, this work presents an open-vocabulary semantic mapping pipeline that integrates TALOS (TAgging–LOcation–Segmentation) with the probabilistic, instance-aware Voxeland framework. TALOS employs a modular generative approach in which vision–language and large language models infer categories for the objects in the scene, an open-vocabulary grounding model localizes their instances, and a category-agnostic segmentation model produces pixel-accurate masks. These predictions are combined with depth data and fused into persistent 3D maps using Voxeland while preserving geometric, semantic, and instance uncertainty. The approach was evaluated on diverse ScanNet v2 scenes and compared with two alternative perception front-ends: the closed-vocabulary Detectron2/Mask R-CNN and the open-vocabulary discriminative model YOLOE. Across five independent runs, TALOS achieves a mean mAP@0.5 of 0.1607±0.0255, compared with 0.0834 for YOLOE and 0.0453 for Detectron2/Mask R-CNN, and exceeds both baselines in every run. Qualitative results further show a better balance between map completeness, geometric clarity, semantic coherence, and instance separation.
Open-vocabulary semantic maps are key to the autonomy of mobile robots in complex, unstructured environments. Compared to traditional techniques, open-vocabulary semantic maps represent a significant improvement, as they are not limited by a predefined set of categories. For their construction, geometric representation...
Jorge Serrano Mesa, Jesús Moncada Ramírez, Pepe Ojeda et al.· Jornadas de Automática· 0 citations
This framework combines simultaneous localization and mapping (SLAM), visual-language feature extraction, incremental semantic and instance label fusion, and spatial graph construction to enable a construction robot navigation framework that supports open-vocabulary language queries.
Charles M. Raines, I. Fernandez, Mandy Sun et al.· Journal of computing in civi...· 0 citations
This work proposes PointPDF V2, a unified framework that integrates open-set recognition (OSS) and incremental learning (IL) into a cohesive pipeline and introduces a more challenging continual OWSS protocol in 3D, where models must simultaneously preserve the known-class performance, acquire new knowledge, and still i...
Jinfeng Xu, Xianzhi Li, Yi-Xue Hao et al.· IEEE Transactions on Pattern...· 0 citations
SceneBench is introduced, a benchmark of 966 photorealistic 3D scenes reconstructed with Gaussian Splatting and densely annotated with hierarchical semantics spanning scenes, rooms, functional areas, object groups, and individual objects that provides a realistic testbed for developing and evaluating models capable of...
Anubhav Khanal, Prabigya Acharya, Roshni Poudel et al.· 0 citations
Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it...
Cigdem Kokenoz, Amir Salarpour, Alkim Domeke et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.