LATTE: Language-Driven Construction of 3D Driving Scenes for Closed-Loop AD Simulation
TL;DR
LATTE is a language-driven framework for constructing explicit 3D driving scenes from open-world scene requests that not only supports open-world asset insertion and more coherent scene layout, but also produces behaviorally effective test cases that measurably alter downstream AD performance.
Abstract
Closed-loop simulation is central to autonomous driving (AD) safety evaluation, yet it remains difficult to turn natural-language requests into valid 3D test scenarios. Existing language-driven methods either operate in pixel space without explicit scene instantiation, or rely on fixed asset banks and coarse object placement that are insufficient for open-world, test-oriented scene construction. We present LATTE, a language-driven framework for constructing explicit 3D driving scenes from open-world scene requests. LATTE decomposes the problem into two coupled parts: Image-Grounded Asset Construction (IGAC), which acquires insertion-ready 3D assets for missing objects from visual evidence, and Structurally Coherent Layout Optimization (SCLO), which turns language-guided initial placements into simulation-suitable layouts under physical and semantic constraints. The resulting edits are written back to a real-scene-reconstructed simulator and evaluated through closed-loop AD rollouts. Experiments show that LATTE not only supports open-world asset insertion and more coherent scene layout, but also produces behaviorally effective test cases that measurably alter downstream AD performance. The project page is at here.