Skip to content

Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation

Sep 2026 · arXiv.org · Vol abs/2609.20822 · 0 citations · 51 references
Computer Science

Abstract

Coding agents have emerged as a promising paradigm for robot manipulation: a language model writes the robot controller as a program, and agents built in this way now operate robots without robot-specific training.Whether this paradigm is also safe, however, has not been asked. We evaluate coding agent under a safety constraint, where each task pairs a manipulation goal with an obstacle the robot must not touch. The agent pursues the goal but collides with the obstacle in most cases, treating task completion as its sole objective while neglecting safety. The agent reasons about the obstacle in its traces, and the prompt already forbids touching it, so neither perception nor instruction is at fault; the fault lies in the planning, where the stated constraint never becomes a priority. By decomposing manipulation into a route phase and a contact-rich moment, we locate the source of the failure. Along the route, the model cannot prioritize the safety constraint, having no notion of a clearing route and none of replanning once a chosen route becomes infeasible. At the contact, it is unaware that contact execution is bounded by the same constraint. To close this gap, we present SafeHarness, which equips the model with two obstacle-aware harnesses that enable it to prioritize the safety constraint. Obstacle-aware route planning grounds the objects as bounding boxes and draws candidate routes over them as sequences of waypoints. The agent then plans a route in advance, verifies it, replans when necessary, and only then executes it. Obstacle-aware contact execution instead selects the contact position so that the contact itself avoids the obstacle. SafeHarness attains 71.9% task success and 87.5% collision avoidance, surpassing the previous SOTA by 6.5% and 27.0%, respectively. These results are $2.3\times$ and $1.5\times$ those of the same agent without harnesses.

View source

Similar papers

#natural language process... Preprint Sep 2026

Agent as Policy for Robotic Manipulation

This work demonstrates that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training and introduces Agent as Policy (AGP), which places task planning and execution under the agent's control.

Mengzhao Jia, Yang Lin, Xi-Xin Zhang et al. · 23 citations · ⚡1
Preprint Aug 2026

Revisiting the"Push-T"Robot Manipulation Task with Agentic Robotics

This short paper revisits the Push-T task in the context of emerging advances in Agentic Robotics where an LLM coding agent -- Claude Code with Fable 5 -- is prompted to create an algorithmic solution that does not require any demonstration data.

Shuang-Yu Xie, Kai-Peng Chen, Ken Goldberg · 0 citations

Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots

This work introduces a unified formalism for proactive robot assistance, organize it into three levels, and provides a framework to address the highest level of unprompted proactive assistance, and presents a method, GAP, that instantiates the framework, learning from passive observation to anticipate user goals and ac...

Maithili Patel, Sonia Chernova · 0 citations
Sep 2026

In-Context Robot Learning with VLM Agents

GPT-Policy is introduced, a general-agent framework for in-context robot learning that integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome.

Dong-Zhou Cheng, Taoran Yi, Ye Fang et al. · 16 citations · ⚡4

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.