Skip to content

NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation

Jun 2026 · arXiv.org · Vol abs/2606.03159 · 14 citations · ⚡ 2 influential
Computer Science

TL;DR

Preliminary results indicate that a world-action model (WAM) post-trained from OmniDreams achieves strong performance on the Physical AI Autonomous Vehicles NuRec dataset, surpassing the VLA-based Alpamayo 1.5 research policy model while using only 1/5 the total parameters.

Abstract

As autonomous vehicle capabilities advance, the safe evaluation of driving policies in long-tail scenarios remains a critical bottleneck. In closed-loop simulation, the driving policy model actively interacts with the environment, where its actions dynamically update the simulator state and directly influence the next set of generated sensor observations. While recent reconstruction-based neural simulators offer photorealism, they are fundamentally constrained by their initial captured data and struggle to generalize to highly dynamic or novel scenes. To overcome these limitations, we introduce OmniDreams, a foundation generative world model mid- and post-trained from the Cosmos diffusion model to autoregressively generate action-conditioned videos in real time. By leveraging the rich visual priors of Cosmos and mid- and post-training on 21k hours of driving scenarios, OmniDreams synthesizes complex, unobserved phenomena that are hard for traditional simulators to capture, such as extreme weather and unpredictable dynamic agent behaviors. Crucially, it autoregressively conditions its photorealistic sensor generation on past frames, the current simulator state, and immediate driving actions. Deployed in a closed-loop system with the Alpamayo 1 policy model and AlpaSim orchestrator, OmniDreams acts as a highly responsive, reactive environment, providing a scalable and comprehensive solution for training and evaluating next-generation autonomous driving policies. We additionally show preliminary results indicating that a world-action model (WAM) post-trained from OmniDreams achieves strong performance on the Physical AI Autonomous Vehicles NuRec dataset, surpassing the VLA-based Alpamayo 1.5 research policy model while using only 1/5 the total parameters. These results highlight the potential for a real-time world model like OmniDreams to also serve as a backbone for policy architectures.

View source

Similar papers

Sep 2026

ZYT-World: A Real-Time Controllable World Model for Closed-Loop Autonomous-Driving Simulation

This work presents ZYT-World, a single architecture that natively generates four fisheye views with field of view>180{\deg} and three pinhole views and a 19M-parameter variational autoencoder decoder decoder, W8A8 quantization, and the inference engine reduce decoding, backbone, and incremental-execution costs, respect...

Bo-Ni Hu, Xiong Wei, Hao-Ming Huang et al. · 1 citation · ⚡1
#artificial intelligence Preprint Sep 2026

Uranus: Building the Next-Generation Simulation Infrastructure for Embodied AI

This work presents Uranus, a data-driven robot simulator built around a joint-trajectory-conditioned autoregressive diffusion model, providing a unified interface for synchronized multi-view generation across diverse robot embodiments and camera configurations.

Wen-Kang Qin, Yu-Kun Zhou, Noah Shen et al. · 0 citations
Sep 2026

RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts

RoXDrive is introduced, a plug-and-play closed-loop RL framework that enables reliable policy optimization by identifying action-faithful world-model rollouts and an Action-Vision Faithfulness Evaluator for inverse dynamics estimation with geometry-aware auxiliary trajectory supervision, enabling long-horizon assessmen...

Hong-Bin Lin, Chao-Da Zheng, Yi-Ming Yang et al. · 1 citation
Preprint Sep 2026

Safety-Critical Scenarios Emerge from Initial Scenes

Safety-critical driving scenario generation has largely focused on manipulating the behavior of surrounding agents while starting from an initial scene from driving data. This assumption can limit the space of discoverable failures, since driving data can provide little opportunity for meaningful interaction. For examp...

Yin Wu, Jia-Rong Wei, Carl Esselborn et al. · 0 citations
Sep 2026

RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors

Closed-loop driving simulation requires rendered observations to remain reliable as the ego vehicle and surrounding actors move beyond their recorded trajectories, exposing views absent from the source log. Existing data-driven simulators reconstruct dynamic actors from sparse observations, which can result in renderin...

Zi-Jun Zhao, Lie-Wen Liao, Kang Shen et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.