Skip to content
Open access

Open-Vocabulary Instance Segmentation for Scene Understanding in Mobile Robots

Sep 2026 · Robotics · 0 citations · 21 references

TL;DR

An open-vocabulary semantic mapping pipeline is presented that integrates TALOS (TAgging–LOcation–Segmentation–Segmentation) with the probabilistic, instance-aware Voxeland framework and shows a better balance between map completeness, geometric clarity, semantic coherence, and instance separation.

Abstract

High-level robotic tasks, such as those involving planning and interaction, demand a certain degree of scene understanding through suitable representations of the environment that enrich geometric information with object-level semantics, commonly referred to as semantic maps. Traditional techniques to build these maps are restricted to working with a set of categories that is fixed during training, which limits the information the resulting maps can contain. To overcome this limitation, this work presents an open-vocabulary semantic mapping pipeline that integrates TALOS (TAgging–LOcation–Segmentation) with the probabilistic, instance-aware Voxeland framework. TALOS employs a modular generative approach in which vision–language and large language models infer categories for the objects in the scene, an open-vocabulary grounding model localizes their instances, and a category-agnostic segmentation model produces pixel-accurate masks. These predictions are combined with depth data and fused into persistent 3D maps using Voxeland while preserving geometric, semantic, and instance uncertainty. The approach was evaluated on diverse ScanNet v2 scenes and compared with two alternative perception front-ends: the closed-vocabulary Detectron2/Mask R-CNN and the open-vocabulary discriminative model YOLOE. Across five independent runs, TALOS achieves a mean mAP@0.5 of 0.1607±0.0255, compared with 0.0834 for YOLOE and 0.0453 for Detectron2/Mask R-CNN, and exceeds both baselines in every run. Qualitative results further show a better balance between map completeness, geometric clarity, semantic coherence, and instance separation.

Read PDF

Similar papers

Open access Sep 2026

Mapeo semántico de vocabulario abierto mediante 3D Gaussian Splatting

Open-vocabulary semantic maps are key to the autonomy of mobile robots in complex, unstructured environments. Compared to traditional techniques, open-vocabulary semantic maps represent a significant improvement, as they are not limited by a predefined set of categories. For their construction, geometric representation...

Jorge Serrano Mesa, Jesús Moncada Ramírez, Pepe Ojeda et al. · 0 citations

ZIVIL: Zero-Shot Incremental Vision-Language Maps and Spatial Graph Representation of Construction Sites

This framework combines simultaneous localization and mapping (SLAM), visual-language feature extraction, incremental semantic and instance label fusion, and spatial graph construction to enable a construction robot navigation framework that supports open-vocabulary language queries.

Charles M. Raines, I. Fernandez, Mandy Sun et al. · 0 citations
Aug 2026

PointPDF V2: A Unified Framework for Continual Open-World 3D Semantic Segmentation.

This work proposes PointPDF V2, a unified framework that integrates open-set recognition (OSS) and incremental learning (IL) into a cohesive pipeline and introduces a more challenging continual OWSS protocol in 3D, where models must simultaneously preserve the known-class performance, acquire new knowledge, and still i...

Jinfeng Xu, Xianzhi Li, Yi-Xue Hao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

SceneBench is introduced, a benchmark of 966 photorealistic 3D scenes reconstructed with Gaussian Splatting and densely annotated with hierarchical semantics spanning scenes, rooms, functional areas, object groups, and individual objects that provides a realistic testbed for developing and evaluating models capable of...

Anubhav Khanal, Prabigya Acharya, Roshni Poudel et al. · 0 citations
Preprint Oct 2026

Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers

Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it...

Cigdem Kokenoz, Amir Salarpour, Alkim Domeke et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.