Skip to content

Dynamic Manipulation with World-Action Models via Counterfactual Planning

Sep 2026 · arXiv.org · Vol abs/2609.33172 · 1 citation · 75 references
Computer Science

TL;DR

DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data and constructs a counterfactual observation that places a predicted target position in a familiar robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery behavior from an unfamiliar robot-target configuration.

Abstract

World-Action models (WAMs) trained on static demonstrations often fail to manipulate moving targets even when they possess the required manipulation skills. We attribute this failure to target-response collapse: as execution advances, the policy becomes increasingly biased toward the learned continuation of its ongoing behavior and less responsive to target relocation. To bridge the gap between what the model has learned and what it can generate from the current context, we formulate dynamic manipulation as counterfactual planning by decoupling the context used for plan generation from the physical state used for execution. Our framework, Dynamic Predictive Planning (DPP), first uses the WAM's predictive rollout to estimate when an interaction is expected to occur, and combines this timing estimate with observed target motion to predict the target's future interaction position. DPP then constructs a counterfactual observation that places this predicted target position in a familiar robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery behavior from an unfamiliar robot-target configuration. The resulting plan is connected to the robot's actual state during execution. DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data. Experiments in simulation and on a real robot demonstrate consistent improvements across diverse target motions, with simulation performance surpassing all evaluated baselines, including methods additionally trained on dynamic data. Project page: https://methoder00.github.io/DPP/

View source

Similar papers

Achieve What You Imagined: Learning to Align Actions with Visual Plans

This work uses a frozen action-conditioned world model to predict action-conditioned consequences and construct feedback based on consistency between the two future predictions and alignment with the terminal goal to show that cross-model prediction discrepancy can provide useful feedback for improving robot policies u...

Yu-Heng Qiao, Zi-Ran Wei, Xiao-Hang Wang et al. · 0 citations
Preprint Aug 2026

HarnessWAM: Bridging Prediction and Deliberation in World Action Models

Results demonstrate that model-external structured state maintenance and closed-loop agentic decision making can effectively extend the local control capabilities of WAMs into embodied task execution that is plannable, verifiable, and recoverable.

Zhao-Peng Gu, Bing-Ke Zhu, Tianxin Lin et al. · 5 citations
Preprint Oct 2026

Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation

Long-horizon compositional manipulation has become increasingly important for real-world robot deployment, where a single task involves multiple coordinated subtasks. Existing world-action models (WAMs) jointly predict short-horizon visual futures and actions, but typically lack explicit subtask-level reasoning. We pro...

Shu-Kai Gong, Xuan-Ran Zhai, Yintianrun Zhang et al. · 2 citations · ⚡1
Preprint Sep 2026

CST-WM: A Causally Structured World Model for Embodied Visual Tracking

Embodied visual tracking requires a robot to choose actions that keep a moving target observable at a suitable distance, and to recover it after occlusion, out-of-view drift, or distractor crossings. We cast the task as planning over future target evidence with an action-conditioned world model. In logged tracking data...

Jun-Yi Hu, Shuaihang Yuan, Jia-Zhao Liang et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Learning Counterfactual World Models for Embodied Reasoning under Partial Observability

This work introduces Counterfactual Latent World Models (CLWM), which combine a recurrent belief-state encoder, action-conditioned latent dynamics, and a contrastive counterfactual objective that separates futures induced by distinct interventions even when their observations look alike.

Todd Y. Zhou, Daniella Zhang · 0 citations

Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation

World-Action Models (WAMs) couple action generation with predictions of how physical interactions unfold. However, current post-deployment learning paradigms typically improve behavior without requiring better world predictions. Especially in dexterous manipulation, small execution errors can compound in high-dimension...

Xiang-Cheng Zhan, Zi-Rui Chen, Yi-Chen Zhao et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.