Skip to content

Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes

Sep 2026 · arXiv.org · Vol abs/2609.36777 · 3 citations · 53 references
Computer Science

TL;DR

This work introduces Code4Scene, a benchmark of 190 Unreal Engine cases built from human-assembled scenes that evaluates coding agents on two complementary settings under a shared execution interface, and exposes a gap between plausible 3D generation and reliable spatial reasoning and state control.

Abstract

Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely control scene state remains unclear. The generated 3D scene is a persistent, executable artifact: a convincing render can hide incorrect spatial relations, intersecting objects, or unintended modifications. We introduce Code4Scene, a benchmark of 190 Unreal Engine cases built from human-assembled scenes that evaluates coding agents on two complementary settings under a shared execution interface. Construction tests scene-level spatial reasoning from open-ended language specifications, where many realizations are valid; editing tests precise control of scene state, where the agent must recover the target scene from reference images while preserving everything else. Rather than scoring code or rendered views, Code4Scene evaluates the generated engine-native scene for task fulfillment, artifact integrity, and static physical validity, with edits additionally compared against withheld ground truth. Across 14 coding-agent configurations on the 95-case public set, construction and editing performance are strongly correlated but not interchangeable (Spearman $\rho = 0.78$): Claude Fable 5.1 leads construction, Gemini 3.8 Flash leads editing, and GPT-6 Astra narrowly leads overall. Spatial Composition is the weakest construction category for every agent, while editing remains imprecise: the best Repair F1 is only 0.527, and 35.8% of edits that fully recover the target still introduce unintended changes elsewhere in the scene. These results expose a gap between plausible 3D generation and reliable spatial reasoning and state control.

View source

Similar papers

Sep 2026

LEGO-Anything: Coding Agents for 3D Scene Reconstruction

These readouts suggest that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images, suggesting that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images.

Xi-Rui Li, Peng Shi, Ming-Wen Dong et al. · 1 citation
#computer vision Review Sep 2026

ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts

ReFigBench, a benchmark and evaluation framework built on 1,000 real overview figures retrieved from arXiv papers with full provenance, is introduced, exposing the tension between fidelity and editability as the central challenge for practical multimodal document agents.

Li-Yang Fan, Chi Wei, Yi-Tai Li et al. · 0 citations
Sep 2026

ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptions

A unified framework for generating physically consistent dynamic 3D scenes from text, with outputs directly executable in Unreal Engine, using the ESG, which specifies entities with physical attributes, spatial relations, and event-driven timelines in a machine-checkable form.

Xin Fang, Zhi-Yuan Fang, Ren-Gan Xie et al. · 0 citations
#artificial intelligence Preprint Aug 2026

PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides

PPTBench establishes a measurable testbed for studying visual coding and advancing agents toward more reliable visual creation, and shows that agents can generally produce valid slide files, but struggle to produce high-quality reconstructions that faithfully recover the semantics and visual structure of the target.

Xiao-Qiu Wang, Yi-Zhe Chi, Wen-Yi Li et al. · 0 citations
Preprint Sep 2026

What Makes a 3D Scene Editable? A Factorized Benchmark of Fidelity, Locality, Consistency, and Preservation

Neural 3D scene editing is often evaluated by semantic alignment alone, although a convincing result may alter unrelated content or become inconsistent across views. We introduce EditBench3D, a representation-agnostic benchmark that treats editing as controlled information replacement. It evaluates four complementary p...

Sariah Patro, Arjun Mehra, Nikhil Bhatia · 0 citations
Preprint Aug 2026

Beyond Placement and Articulation: Usage-Driven Code Scenes for Embodied Interaction

RoomWright is presented, an agentic usage-driven framework for generating 3D scenes represented entirely as code for embodied interaction, providing interactive environments for embodied AI and policy learning.

Zijian Xiao, Zi-Peng Ye, Jin-Kun Hao et al. · 2 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.