Skip to content

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

Sep 2026 · arXiv.org · Vol abs/2609.38059 · 1 citation · 39 references
Computer Science

TL;DR

WorldLine is introduced, an action-driven visual simulator that decouples transferable dynamics learning from heterogeneous action grounding and makes WorldLine a scalable and efficient visual simulator for policy evaluation and embodied planning.

Abstract

Real-world robot learning is constrained by the cost of collecting experience and evaluating candidate behaviors. Video generation models offer a scalable foundation for visual simulators that predict action outcomes before physical execution. Yet they often favor visual plausibility over accurate action following and coherent robot--object dynamics, while action-conditioned simulators depend on scarce, embodiment-specific data that are difficult to share across incompatible control spaces. We introduce WorldLine, an action-driven visual simulator that decouples transferable dynamics learning from heterogeneous action grounding. WorldLine learns manipulation dynamics from more than 10,000 hours of action-free robot videos and grounds them using over 2,000 hours of action trajectories across more than ten embodiments. An image-space action representation provides a shared control interface across embodiments, while multi-view and failure-enriched training with relational regularization improves interaction-sensitive prediction. Robot-focused few-step distillation enables efficient causal rollout while preserving action-critical motion. Across held-out and out-of-domain settings, WorldLine maintains strong visual quality and robot-motion agreement; on failed trajectories, it improves robot-mask IoU by 0.1626 over the strongest baseline. It predicts trajectory success with 74% mean accuracy across RoboTwin and AgiBot, one percentage point above the strongest baseline. Without RoboTwin training or adaptation, its rollouts improve task success by up to 21.4 percentage points over direct policy execution. Together, these capabilities make WorldLine a scalable and efficient visual simulator for policy evaluation and embodied planning. More results are available at \href{https://zhengsh123.github.io/WorldLine/}{project page}.

View source

Similar papers

Preprint Oct 2026

IronMan: Information-Constrained Video-Action Learning for Robot Manipulation

Video Action Models (VAMs) couple visual dynamics modeling with action generation for robot manipulation. However, video representations are not naturally suited to action generation, as exposing the action policy to excessive visual detail can impair its generalization ability. Therefore, we introduce IronMan (Informa...

Yuan-Shuo Zhang, Wen-Zhe Zhao, Zi-Xing Lei et al. · 0 citations
Preprint Oct 2026

Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics

Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions. However, existing robotic datasets provide limited coverage of the diverse interaction dynamics required for generalizable world modeling. Large-scale egocentric human videos offer...

Pei-Cheng Tu, Cong Wang, Han-Xin Zhu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning

Vision-Language-Action (VLA) models have made substantial progress in robotic manipulation, but their generalization remains constrained by the diversity and coverage of action-labeled robot demonstrations. Human videos provide much broader behavioral and environmental experience, while learning from them faces a funda...

Chaehyun Kim, Chi-Pin Huang, Jisoo Kim et al. · 0 citations

Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation

World-Action Models (WAMs) couple action generation with predictions of how physical interactions unfold. However, current post-deployment learning paradigms typically improve behavior without requiring better world predictions. Especially in dexterous manipulation, small execution errors can compound in high-dimension...

Xiang-Cheng Zhan, Zi-Rui Chen, Yi-Chen Zhao et al. · 1 citation
Sep 2026

Rethinking Representations for World-Action Modeling

World-action models jointly learn robot policies and predict future observations, making the representation space an interface between control and prediction. We study the design of this space through controlled comparisons, finding that neither reconstruction fidelity nor pre-trained perceptual features alone ensure e...

Hao-Yi Jiang, Liu Liu, Xin-Jiang Wang et al. · 5 citations
#artificial intelligence Preprint Oct 2026

SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models

Action-conditioned robot world models must respond precisely to robot trajectories while preserving realistic visual dynamics, yet learning both from heterogeneous robot videos remains challenging. Simulation offers structured motion supervision, but appearance differences hinder direct transfer, and inaccurate simulat...

Xiao-Dong Wang, Tian-Le Li, Chuan-Xin Song et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.