Skip to content
Book Open access

LATTE: Language-Driven Construction of 3D Driving Scenes for Closed-Loop AD Simulation

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · pp. 3279-3288 · 0 citations

Abstract

Closed-loop simulation is central to autonomous driving (AD) safety evaluation, yet it remains difficult to turn natural-language requests into valid 3D test scenarios. Existing language-driven methods either operate in pixel space without explicit scene instantiation, or rely on fixed asset banks and coarse object placement that are insufficient for open-world, test-oriented scene construction. We present LATTE, a language-driven framework for constructing explicit 3D driving scenes from open-world scene requests. LATTE decomposes the problem into two coupled parts: Image-Grounded Asset Construction (IGAC), which acquires insertion-ready 3D assets for missing objects from visual evidence, and Structurally Coherent Layout Optimization (SCLO), which turns language-guided initial placements into simulation-suitable layouts under physical and semantic constraints. The resulting edits are written back to a real-scene-reconstructed simulator and evaluated through closed-loop AD rollouts. Experiments show that LATTE not only supports open-world asset insertion and more coherent scene layout, but also produces behaviorally effective test cases that measurably alter downstream AD performance. The project page is at here.

Read PDF

Similar papers

Sep 2026

ZYT-World: A Real-Time Controllable World Model for Closed-Loop Autonomous-Driving Simulation

This work presents ZYT-World, a single architecture that natively generates four fisheye views with field of view>180{\deg} and three pinhole views and a 19M-parameter variational autoencoder decoder decoder, W8A8 quantization, and the inference engine reduce decoding, backbone, and incremental-execution costs, respect...

Bo-Ni Hu, Xiong Wei, Hao-Ming Huang et al. · 1 citation · ⚡1
Preprint Oct 2026

FlowSim: Rendering-Free Closed-Loop Simulation for Evaluating End-to-End Driving Systems

This paper proposes FlowSim, a rendering-free simulation paradigm that predicts the internal latent representation directly from a previous latent and target-frame symbolic scene conditions, and establishes FlowSim as an efficient and practical alternative to conventional image-space simulation methods.

Han-Feng Wu, Alvaro Marcos-Ramiro, Michael Schmidt et al. · 0 citations
Book Open access Nov 2025

ArtiWorld: LLM-Driven Articulation of 3D Objects in Scenes

Building interactive simulators and scalable robot-learning environments requires a large number of articulated assets. However, most existing 3D assets used in simulation are rigid, and manually converting them into articulated objects with executable kinematic structures is highly labor-intensive, time-consuming, and...

Yi-Xuan Yang, Lu-Yang Xie, Zhen Luo et al. · 4 citations
#artificial intelligence Preprint Oct 2026

Odyssey: A Closed-Loop Benchmark for Long-Horizon Real-World Driving with Explicit Navigation Routes

Closed-loop evaluation of end-to-end driving requires continuous rollouts that reveal how earlier decisions affect subsequent driving. However, existing benchmarks evaluate only short segments and fail to capture later consequences. Ambiguous directional commands also obscure the intended navigation objective. We intro...

Jungho Kim, Hongjae Shin, Seung-Su Yu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

SpaceAct: Benchmarking 3D Spatial and Temporal Reasoning via Closed-Loop Action and Active Perception

To solve problems in 3D space rather than merely describe images, multimodal large language models (MLLMs) need 3D spatiotemporal reasoning: reconstructing 3D geometry from occluded 2D views, anticipating physical dynamics, and executing precise, closed-loop actions. Prior studies test passive VQA on static images or s...

Cheng-Long Shi, Ze-Kai Yuan, Yue Song et al. · 0 citations
Sep 2026

RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors

Closed-loop driving simulation requires rendered observations to remain reliable as the ego vehicle and surrounding actors move beyond their recorded trajectories, exposing views absent from the source log. Existing data-driven simulators reconstruct dynamic actors from sparse observations, which can result in renderin...

Zi-Jun Zhao, Lie-Wen Liao, Kang Shen et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.