Skip to content

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

Sep 2026 · 4 citations · 39 references
Computer Science

TL;DR

World Action Agent is presented, a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone.

Abstract

General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.

View source

Similar papers

Sep 2026

AR-WAM: A Visual-Conditioned Agent-Ready World Action Model for Robotic Manipulation

AR-WAM is presented, a visual-conditioned, agent-ready world action model that replaces language with two complementary conditions: a visual grounding prompt and a learnable operation token dictating the atomic skill to execute, predicting scene evolution within compact latent states while decoding actions.

Yi-Cheng Jiang, Ze-Sen Gan, Xiao-Bo Wang et al. · 3 citations
Sep 2026

Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models

FIND is introduced, an agentic real-world RL framework that closes the loop between scene understanding, weakness-aware practice, self-evaluation, and policy improvement in a persistent workspace and reframes autonomous practice as a scene-conditioned, performance-aware task-selection problem.

Yuan Fang, Ze-Chu Li, Hao-Lei Tong et al. · 3 citations · ⚡1
Preprint Sep 2026

HINT: Human-Intent Inception for Long-Horizon Robot Manipulation

HINT (Human-INTent INcepTion), an agentic framework inspired by the human manipulation principles: semantic intent changes sparsely at manipulation-pattern transitions, whereas continuous control primarily depends on the evolving object-hand relationship.

Ming-Yu Mei, Haojie Xu, Shi-Hao Jin et al. · 0 citations
#artificial intelligence Preprint Sep 2026

2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation

2AM shows that task memory can remain Agent-side and shows that Action Model capability depends not only on what the policy has learned, but on how precisely the Agent can steer it, and isolates this question through a deliberately constrained design.

Yu-Tong Hu, Feng Chen, Xue-Zhi Cao et al. · 2 citations · ⚡2
Preprint Sep 2026

Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies

This work adds one feature-alignment term to ordinary VLA training: a frozen world model is run over the training frames once and cached, and the student learns to agree with that cache, indicating a broad representational prior rather than a fragile alignment between two particular networks.

T. Dao, Sankalp Yamsani, Jaden Park et al. · 1 citation
Sep 2026

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness

This work introduces HarnessVLN, a zero-shot, training-free framework that unifies instruction-following and object-goal navigation through a shared Agent Harness and demonstrates its applicability to both navigation tasks in real-world environments.

Yang Chen, Li-Rong Che, Zhen-Yu Huang et al. · 8 citations · ⚡2

Related blog posts

Hugging Face Blog Nov 10, 2026

Decision Models in Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.