Skip to content

RoGSW4RLD: Feed-Forward 4D Gaussian Lifting for Robot World Model Rollouts

Sep 2026 · arXiv.org · Vol abs/2609.35311 · 0 citations · 36 references
Computer Science

TL;DR

RoGSW4RLD is introduced, a feed-forward framework that lifts synchronized multi-camera rollouts into a unified, time-queryable metric 4D Gaussian field and directly reconstructs the visual future generated by existing world models.

Abstract

Action-conditioned video world models predict future robot interactions from multiple cameras, yet their outputs remain disparate video collections rather than a shared metric scene queryable across viewpoints and time. While existing 4D reconstruction methods offer a path to spatialize these predictions, independently reconstructing and merging each camera stream fails to enforce cross-view consistency. This limitation is particularly detrimental when combining moving robot-mounted cameras with fixed external views. To address this, we introduce RoGSW4RLD, a feed-forward framework that lifts synchronized multi-camera rollouts into a unified, time-queryable metric 4D Gaussian field. Rather than learning a separate geometric transition model, RoGSW4RLD directly reconstructs the visual future generated by existing world models. Its core innovation is a two-stage architecture: Stage 1 jointly forms the metric 4D field by fusing cross-view evidence with robot-specific articulated geometry and kinematics, while Stage 2 refines the field's geometry and appearance while strictly preserving the initial temporal displacements. Evaluated on 256 held-out DROID episodes, RoGSW4RLD significantly outperforms camera-wise reconstruction with calibrated merging, improving novel-view PSNR by 2.15 dB, reducing depth AbsRel by 47%, and lowering robot displacement error by 61%. These robust gains extend to action-conditioned Cosmos 3 rollouts, demonstrating that predicted video futures can be successfully translated into consistent, spatially queryable 4D metric representations.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

DepthWorld: 3D World Model for Robot Manipulation

This work introduces a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics, and trains DepthWorld, a Stable Video Diffusion-based world model that jointly...

Jai Bardhan, Josef Sivic, Vladimír Petrík · 2 citations
Preprint Aug 2026

UniQuery4R: Unified 4D Scene Reconstruction from a Single Query

UniQuery4R is presented, a query-conditioned framework that encodes a multi-frame clip once and selects the source view, target view, and continuous source-image coordinate only at decoding time via source-to-target cross-attention, and introduces a direction-magnitude parameterization of scene flow with separate super...

Tiancheng Chen, Sheng Tang, Wenhua Jin et al. · 1 citation
Preprint Aug 2026

GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation

GaussianDream demonstrates that training-time current Gaussian reconstruction and future Gaussian prediction provide effective 3D supervision, but its dense VGGT/TGE-based prefix jointly carries state, dynamics, and action-conditioning information.

Yu-Qing Jiang, Zi-Jian Zhang, Wei-Tao Zhou et al. · 1 citation
Preprint Aug 2026

GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models

GaussianWAM is proposed, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field and improves performance on standard LIBERO and shows positive transfer trends on RoboTwin and real-world manipulation.

Zi-Jian Zhang, Yu-Qing Jiang, Wei-Tao Zhou et al. · 3 citations
Preprint Sep 2026

S4VY: Segment Anything in Feed-Forward 4D Visual Geometry

Accurate instance segmentation in dynamic scenes is important for downstream applications such as robotics and autonomous driving. Existing Segment Anything models operate primarily on 2D image or video masks and preserve identity through sequential memory, while promptable 4D instance segmentation built upon feed-forw...

Jingdong Zhang, Xin Li, Jan Kautz et al. · 1 citation
#artificial intelligence Preprint Sep 2026

DiffWAM: A Fast and Efficient Navigation World Action Model

DiffWAM is presented, a geometry-conditioned navigation world-action model that directly transforms multi-level predictive features into continuous camera trajectories, providing a direct alternative to generate-then-reconstruct navigation pipelines.

Morui Zhu, Yu-Ze Wu, Xi-Jie Huang et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.