Skip to content

VPTwin: Real-Sim-Real Video Prediction for Robotic Manipulation Planning

Sep 2026 · 0 citations · 59 references
Computer Science

TL;DR

VPTwin is proposed, a Real-Sim-Real video prediction framework that anchors real-world future prediction using real-synchronized simulation twins and establishes a predictive planning loop using VPTwin to visually verify VLM-proposed actions and guide reliable real-world execution.

Abstract

While action-conditioned video prediction provides an intuitive world model for robotics, purely data-driven predictors often suffer from compounding errors and physically implausible hallucinations in long-horizon rollouts, severely undermining downstream action planning. We propose VPTwin, a Real-Sim-Real video prediction framework that anchors real-world future prediction using real-synchronized simulation twins. For a target manipulation task, a VLM reconstructs an executable digital twin from a real demonstration episode. To accommodate the ill-posed estimation of unobserved physical properties, Isaac Sim simulates multiple forward dynamic rollouts across randomized physical configurations under candidate action trajectories. Using these rollouts as in-context references, VPTwin harmonizes both domains, using simulation dynamics to enforce physical plausibility while capturing unmodeled contact interactions from real video. Furthermore, we establish a predictive planning loop using VPTwin to visually verify VLM-proposed actions and guide reliable real-world execution. Evaluations show substantial reductions in physical hallucinations during video prediction and marked improvements in manipulation planning performance.

View source

Similar papers

Sep 2026

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

WorldLine is introduced, an action-driven visual simulator that decouples transferable dynamics learning from heterogeneous action grounding and makes WorldLine a scalable and efficient visual simulator for policy evaluation and embodied planning.

Shenghe Zheng, Wen-Bo Li, Ji-Yao Zhang et al. · 1 citation
Preprint Oct 2026

SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models

Driving world action models (WAMs) learn from paired trajectories and future observations. Post-training with reinforcement learning allows the policy to explore alternative trajectories and improve planning using driving rewards. However, driving logs contain future videos only for the recorded trajectories, leaving t...

Chang-Hui Jing, Hao-Chen Tian, Hong-Chen Li et al. · 1 citation
#artificial intelligence Preprint Oct 2026

SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models

Action-conditioned robot world models must respond precisely to robot trajectories while preserving realistic visual dynamics, yet learning both from heterogeneous robot videos remains challenging. Simulation offers structured motion supervision, but appearance differences hinder direct transfer, and inaccurate simulat...

Xiao-Dong Wang, Tian-Le Li, Chuan-Xin Song et al. · 0 citations
Preprint Oct 2026

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos

Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manual environment construction and calibration. We study whether frontier foundation models and coding agents can automate this process end to end. We formulate \emph{auto...

Jin-Zhou Tang, Z. Zhang, Jing Yang et al. · 2 citations
Preprint Oct 2026

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

Results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer, and shows that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points.

Huang Huang, Wen-Si Ai, Zi-Yu Chen et al. · 0 citations
#artificial intelligence Preprint Oct 2026

World Action Modeling with Progressive Visual Planning

World action models (WAMs) have emerged as a promising paradigm for robotic control by jointly predicting future visual dynamics and actions from an initial observation and instruction. However, existing WAMs struggle with long-horizon prediction, as generating dense video rollouts is highly inefficient. Some recent WA...

Fei Zhang, Zhao-Chong An, D. Frost et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 13, 2026

Helping robots fit into the human workplace

Founded by Anthony Jules ’92, SM ’92 and Professor Emeritus Rodney Brooks, Robust AI’s robots are designed to collaborate with humans in factories and warehouses.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.