Skip to content

ZYT-World: A Real-Time Controllable World Model for Closed-Loop Autonomous-Driving Simulation

Sep 2026 · arXiv.org · Vol abs/2609.21712 · 1 citation · ⚡ 1 influential · 56 references
Computer Science

TL;DR

This work presents ZYT-World, a single architecture that natively generates four fisheye views with field of view>180{\deg} and three pinhole views and a 19M-parameter variational autoencoder decoder decoder, W8A8 quantization, and the inference engine reduce decoding, backbone, and incremental-execution costs, respectively.

Abstract

Generative world models offer controllable and repeatable closed-loop simulation for end-to-end and vision-language-action driving policies, but production deployment exposes three unresolved requirements: faithfully reproducing a mixed fisheye-pinhole rig at native resolutions; reconciling causal, per-timestep interaction with long-horizon stability and low latency; and preserving scene identity when a location is revisited. We present ZYT-World, a single architecture that natively generates four fisheye views with field of view>180{\deg} and three pinhole views. Projection-specific Plucker adapters encode camera geometry, ego-motion adaptive layer normalization provides global motion control, and a lightweight pixel-aligned layout conditions traffic participants and signals through instance-level boxes, headings and colors. Heterogeneous training combines full-rig geometric coverage with high-resolution detail. Teacher forcing, causal consistency distillation, self-rollout distribution matching distillation, and RigCritic transform a 40-step bidirectional teacher into a one-step, per-latent streaming generator, with RigCritic evaluating the seven-view rig jointly. A 19M-parameter variational autoencoder decoder (TinyVAE), W8A8 quantization, and our inference engine reduce decoding, backbone, and incremental-execution costs, respectively. Finally, cross-trajectory pairs derived from real captures train a plug-in implicit-memory module that preserves place-specific evidence. On the internal multi-view test set, the one-step model retains more than 90% of the teacher's PSNR and SSIM, while FID, FVD, and LPIPS stay within 11% of the teacher. Under the generator-only timing in Figure 2, it is 107.7 times faster than the 40-step bidirectional teacher. TinyVAE decodes 59.8 times faster than Wan. 30s rollouts and cross-trajectory revisits show the intended long-horizon and memory behavior.

View source

Similar papers

Astronex-World 1.0: Real-Time Interactive World Model Foundation

Astronex-World 1.0 is presented, an open controllable video world-model foundation that provides a bidirectional model for full-context generation and a causal model with block-causal attention and cross-block KV caching for persistent generation, both built on the Wan2.2-TI2V-5B prior.

Xin Zhou, Cong-Wen Miao · 3 citations · ⚡1
Preprint Oct 2026

FlowSim: Rendering-Free Closed-Loop Simulation for Evaluating End-to-End Driving Systems

This paper proposes FlowSim, a rendering-free simulation paradigm that predicts the internal latent representation directly from a previous latent and target-frame symbolic scene conditions, and establishes FlowSim as an efficient and practical alternative to conventional image-space simulation methods.

Han-Feng Wu, Alvaro Marcos-Ramiro, Michael Schmidt et al. · 0 citations
#artificial intelligence Preprint Oct 2026

TerraTransfer++: Demonstration-Free End-to-End Planner

End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all existing paradigms: imitation learning requires fleet-scale driving logs whose behavior distribution only thinly covers the safety-critical tai...

Zikang Xiong, Wei-Xin Li, Zhouchonghao Wu et al. · 0 citations
Book Open access Nov 2026

LATTE: Language-Driven Construction of 3D Driving Scenes for Closed-Loop AD Simulation

Closed-loop simulation is central to autonomous driving (AD) safety evaluation, yet it remains difficult to turn natural-language requests into valid 3D test scenarios. Existing language-driven methods either operate in pixel space without explicit scene instantiation, or rely on fixed asset banks and coarse object pla...

Zai-Yi Hu, Hai-Feng Wu, Yi-Yi Liao et al. · 0 citations
Preprint Oct 2026

UniWorld: A Unified World Model for Real-Time Visual World Simulation

Continuous visual world simulation requires coupling evolving scene geometry with temporally persistent appearance. We present UniWorld, a unified framework that integrates depth trajectories and reference appearance within a shared video generation model. The central design maintains visual and geometric histories on...

Xin Zhang, Ya-Bo Chen, Shuai-Wei Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

DepthWorld: 3D World Model for Robot Manipulation

This work introduces a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics, and trains DepthWorld, a Stable Video Diffusion-based world model that jointly...

Jai Bardhan, Josef Sivic, Vladimír Petrík · 2 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.