Aug 2026· IEEE International Symposium on Robot and Human Interactive Communication· pp. 2038-2045· 0 citations· 24 references
Computer Science
TL;DR
A distributed conversational framework for human-robot collaborative manipulation that integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack, enabling flexible deployment across distributed hardware while maintaining a responsive control loop.
Abstract
This paper presents a distributed conversational framework for human-robot collaborative manipulation that integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack. Language understanding, visual grounding, orchestration, and motion execution run as separate ROS 2 nodes, enabling flexible deployment across distributed hardware while maintaining a responsive control loop. From free-form user commands, the system generates structured action requests for pick, place, and handover. It uses a VLM to return image-space targets, which are converted into metric robot-frame goals using depth and calibration. A web dashboard exposes intermediate intent and grounding overlays (pixel, depth, and robot-frame) and requires explicit operator confirmation before any motion is executed. Experiments on a Franka FR3 platform evaluate task reliability and latency in a dice-based tabletop manipulation scenario under increasing scene ambiguity, comparing alternative LLM/VLM configurations. Code and full documentation are available at github.com/cogrob-tuni/franka-llm.
Robots that follow natural-language instructions unlock a wide range of autonomous applications in households, warehouses, and labs. Real-world environments are diverse and often heavily cluttered, requiring robots to plan over multiple steps, act, and replan as they work toward a goal. We present CLARO (CLutter-Aware...
Sungsu Kim, Vineet Bhat, P. Krishnamurthy et al.· 1 citation· ⚡1
This work proposes a non-invasive framework that connects LLM reasoning with ROS-based navigation through a navigation-oriented representation layer, exposed through the Model Context Protocol as standardized, reusable tools so that any MCP-compatible LLM can access them without robot-specific wrappers.
Jungsoo Lee, Jaegyun Park, Wansoo Kim· arXiv.org· 0 citations
This work introduces DuoMind, a distributed hierarchical framework for multi-robot coordination through semantic communication, and develops RoboPoly, a benchmark comprising long-horizon manipulation tasks that require coordinated, closed-loop execution under distributed control.
Han-Chu Zhou, De-Chen Gao, Hang Wang et al.· 1 citation
AURaPath is extended, an AR system for interactive 6-DoF waypoint determination on a UR10 collaborative robot using HoloLens2, with a voice modality for hands-free authoring, and the ability to decline out-of-scope input determines whether a model is suitable for the language-mapping tier of the safety-gated architectu...
M. Hashemi, Kasra Mojallal, Saeed Mozaffari et al.· Applied Sciences· 0 citations
CognitiveReality turns a robot's RGB-D stream into a live, semantically indexed Gaussian-TSDF map shared by an operator in virtual reality and a tool-using language agent, and raises object quality by 2-5 dB on robot data.
Timofei Kozlov, Д. А. Малюков, A. Marchenko et al.· arXiv.org· 0 citations
This paper presents a ROSGPT-based semantic navigation system for multi-user autonomous mobile robot (AMR) operation under ROS 2. An environment-specific prompt converts Traditional Chinese, English, and mixed-language requests into structured MOVE, ROTATE, and GOTO commands. JSON-format constraints, station-name white...
Sheng-Yung Peng, Jin-Siang Shaw· WSEAS transactions on system...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.