This work introduces RoboWorM, a comprehensive, deployment-aware benchmark that leverages known hardware priors such as robot kinematic trees and fixed camera configurations, to systematically examine structural and spatiotemporal fidelity, and conducts the first large-scale audit of the quality-efficiency trade-off.
Abstract
Video world models (VWMs) are rapidly emerging as scalable neural simulators for robotic manipulation. However, most benchmarks only evaluate them as open-world video generators, using Vision-Language Models (VLMs) that lack strict 3D spatial and kinematic reasoning. To bridge this gap, we introduce RoboWorM, a comprehensive, deployment-aware benchmark that leverages known hardware priors such as robot kinematic trees and fixed camera configurations, to systematically examine structural and spatiotemporal fidelity. RoboWorM curates 1,605 episodes across single-arm, bimanual, and humanoid morphologies, spanning both Markovian and long-horizon non-Markovian tasks under single- and multi-view setups. To explicitly measure physical consistency, we propose two deterministic, VLM-free metrics: KIVA for intra-view kinematic stability, and RIGIS for multi-view geometric consistency. By exploiting intrinsic robotic invariants, KIVA improves hallucination detection alignment with human annotations by 25.8% relative to the strongest frontier VLMs, while RIGIS achieves a 130% higher Spearman correlation with human preferences. Furthermore, since real-world deployment requires low inference latency, we utilize RoboWorM to conduct the first large-scale audit of the quality-efficiency trade-off. Evaluating 9 state-of-the-art acceleration techniques across 3 leading world models (over 29,000 generated rollouts), we demonstrate how these acceleration methods can retain superficial visual quality while silently compromising deployment-critical physical constraints. Using our metrics, we achieve 2.3$\times$ maximum inference speed-up with minimal quality loss.
This work introduces a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics, and trains DepthWorld, a Stable Video Diffusion-based world model that jointly...
Jai Bardhan, Josef Sivic, Vladimír Petrík· 2 citations
TRWORLDBENCH is introduced, a benchmark for evaluating embodied world models through synchronized head, left-wrist, and right-wrist videos and uses 19 metrics to assess tri-view consistency, task alignment, physical and 3D coherence, motion quality, temporal consistency, and visual quality.
Xuan-Yi Liu, Hao-Feng Wang, Rui-Qi Li et al.· 2 citations· ⚡1
Robotic foundation models achieve impressive performance on standard manipulation benchmarks, yet these evaluations typically assume clean, timely, and consistent visual observations throughout execution. We introduce LIBERO-VPro, a benchmark for systematically evaluating the closed-loop visual robustness of robotic fo...
Hui-Qiong Li, Zhi-Ting Mei, Anirudha Majumdar et al.· 1 citation
This work introduces PhysicsLENS, a dataset and benchmark for evaluating plausibility of physical properties grounded in robotics, and shows that plausible-looking videos often ignore the stated property, and that stating the property lowers plausibility only slightly and not significantly.
Isaiah Milkey, Som Sagar, Aditya Taparia et al.· 0 citations
Egocentric human video offers a scalable data source for dexterous manipulation, yet using it to train humanoid robots presents two challenges: (1) an embodiment gap, as human hands differ structurally from robot end-effectors and low-cost egocentric recordings lack the torso kinematics required by conventional retarge...
Hui-Min Pan, Y. Ren, Kun-Peng Song et al.· arXiv.org· 0 citations
WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods, and is introduced as a generative world model that jointly predicts future video and query-based 3D trajectories.
Yu-Hao Lu, Jiong-Xu Chen, He-Xiang Wei et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.