Skip to content
Book Open access

LATTE: Language-Driven Construction of 3D Driving Scenes for Closed-Loop AD Simulation

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 45 references

TL;DR

LATTE is a language-driven framework for constructing explicit 3D driving scenes from open-world scene requests that not only supports open-world asset insertion and more coherent scene layout, but also produces behaviorally effective test cases that measurably alter downstream AD performance.

Abstract

Closed-loop simulation is central to autonomous driving (AD) safety evaluation, yet it remains difficult to turn natural-language requests into valid 3D test scenarios. Existing language-driven methods either operate in pixel space without explicit scene instantiation, or rely on fixed asset banks and coarse object placement that are insufficient for open-world, test-oriented scene construction. We present LATTE, a language-driven framework for constructing explicit 3D driving scenes from open-world scene requests. LATTE decomposes the problem into two coupled parts: Image-Grounded Asset Construction (IGAC), which acquires insertion-ready 3D assets for missing objects from visual evidence, and Structurally Coherent Layout Optimization (SCLO), which turns language-guided initial placements into simulation-suitable layouts under physical and semantic constraints. The resulting edits are written back to a real-scene-reconstructed simulator and evaluated through closed-loop AD rollouts. Experiments show that LATTE not only supports open-world asset insertion and more coherent scene layout, but also produces behaviorally effective test cases that measurably alter downstream AD performance. The project page is at here.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.