Skip to content

CognitiveReality: Robot-Agnostic Semantic Gaussian Mapping with an LLM Agent for Immersive Collaborative VR Teleoperation

Sep 2026 · arXiv.org · Vol abs/2609.31418 · 0 citations · 27 references
Computer Science

TL;DR

CognitiveReality turns a robot's RGB-D stream into a live, semantically indexed Gaussian-TSDF map shared by an operator in virtual reality and a tool-using language agent, and raises object quality by 2-5 dB on robot data.

Abstract

A photorealistic 3D view tells a teleoperator where a robot is, but not what the scene contains, how well each object has been observed, or how to turn pointing and speech into robot action. CognitiveReality turns a robot's RGB-D stream into a live, semantically indexed Gaussian-TSDF map shared by an operator in virtual reality and a tool-using language agent. One mapper binary serves any platform through configuration alone: it ingests poses from robot SLAM, joint kinematics, motion capture or an inline visual tracker, bridges localization outages with a shadow tracker and keyframe-anchored PnP, and maintains open-vocabulary instance identities with per-object quality at 2 Hz. Speech and controller rays are grounded against persistent scene objects through validated typed tools and operator-confirmed robot actions. In the controlled agent evaluation, the deployed local Qwen3-VL-8B router reaches 81.24\% tool exact match, while merge-aware replay correctly redirects 101 absorbed object identifiers. On robot data CognitiveReality exceeds a Gaussian-plus-SDF baseline by 2-8 dB; pose error through 5-40 s SLAM outages stays within 1-8 cm. Deployed live on two quadrupeds, the agent executed 26 of 30 navigation requests and 20 of 20 re-observation requests, raising object quality by 2-5 dB.

View source

Similar papers

Preprint Sep 2026

RoboFind: Multi-Agent Personalized Object Search for People Who Are Blind or Have Low Vision

Blind and low-vision users often need to locate a specific personal object rather than an arbitrary instance of the same category. The task calls for a robot that can move through the space and reach viewpoints the user cannot, and for an accessible interface where the user says which object is meant and learns whether...

Rui-Ping Liu, Shaofang Quan, Qian-De Yin et al. · 0 citations
Sep 2026

DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Library

Human videos offer a scalable source of demonstrations for dexterous robot manipulation. However, existing human-to-simulation-to-robot (Human2Sim2Robot) pipelines rely on predefined procedures that struggle to accommodate diverse object properties and interactions, particularly those involving articulated and deformab...

You-Hui Wang, Yun-Zhu Li, Fei-Fei Li et al. · 4 citations
Preprint Oct 2026

CLARO: Clutter Aware Robot Orchestrator for Open-Vocabulary Grasping using VLM Agents

Robots that follow natural-language instructions unlock a wide range of autonomous applications in households, warehouses, and labs. Real-world environments are diverse and often heavily cluttered, requiring robots to plan over multiple steps, act, and replan as they work toward a goal. We present CLARO (CLutter-Aware...

Sungsu Kim, Vineet Bhat, P. Krishnamurthy et al. · 1 citation · ⚡1
Sep 2026

Finder: Agentic Closed-Loop Object Finding for Embodied Grounding

Finder is presented, an agentic closed-loop object-finding primitive for embodied grounding that maintains a typed loop state that links query-conditioned planning, scoped evidence gathering, candidate verification, and accept/continue/abort control.

Shi-Xiong Xu, Zhi-Yuan Chen, Song Ding et al. · 0 citations
#artificial intelligence Review Sep 2026

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

World Action Agent is presented, a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone.

Ye-Hang Zhang, Hao-Jian Huang, Yi-Fan Chang et al. · 4 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.