Skip to content

LEGO-Anything: Coding Agents for 3D Scene Reconstruction

Sep 2026 · arXiv.org · Vol abs/2609.36380 · 1 citation · 51 references
Computer Science

TL;DR

These readouts suggest that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images, suggesting that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images.

Abstract

A 3D scene reconstructed from a single image is most useful when represented not as a rendering or a fixed 3D output, but as an explicit scene program whose execution yields a scene that can be inspected, edited, and queried. We present LEGO-Anything, an Image-to-Code framework in which a coding agent iteratively writes and executes Blender code, inspects scenes and renderings, and revises the program. To evaluate end-to-end scene recovery, we introduce LEGO-Bench, a simulator-grounded benchmark with 208 images from 104 diverse indoor and outdoor scenes. LEGO-Bench separately scores artifact validity, visible-surface geometry, and rendered appearance. Its simulator-grounded design enables extensibility and precise automatic evaluation. Among evaluated agents, GPT-6-astra achieves the strongest overall results, with 53.4% indoor and 39.6% outdoor scores, yet substantial gaps remain between delivering valid scene artifacts and faithfully recovering scene geometry and appearance. Analysis of agent construction trajectories reveals three recurring issues: weak scene initialization, regressive edits during iteration, and unreliable self-evaluation. These findings motivate LEGO-Plugin, a training-free harness plugin for more controlled iterative scene construction, which improves all six evaluated models, with relative gains of up to 62.7% in overall score. Finally, we test whether reconstructed scenes can represent natural images and support vision tasks. In LEGO-World, we derive object detections, instance masks, and relative depth as deterministic queries on scenes reconstructed by GPT-6-astra. These readouts show non-trivial performance across all three tasks but fall well short of specialized vision models, suggesting that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images.

View source

Similar papers

Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes

This work introduces Code4Scene, a benchmark of 190 Unreal Engine cases built from human-assembled scenes that evaluates coding agents on two complementary settings under a shared execution interface, and exposes a gap between plausible 3D generation and reliable spatial reasoning and state control.

Xiao-Kang Ye, Siddhant Hitesh Mantri, Zi-Meng Chen et al. · 4 citations
Sep 2026

Fysiverse-3D-SimReady Technical Report: Agentic Physical Simulation for Pragmatic 3D World Reconstruction

Experiments show that Fysiverse-3D-SimReady improves input-view alignment, contact plausibility, and physical stability over existing single-image reconstruction and scene generation baselines, while enabling goal-conditioned physical interactions from a single image.

Lin-Tao Wang, Ming-Yang Sun, Yang Liu et al. · 1 citation
Preprint Oct 2026

Tetris3D: 3D Scene Generation With Objects That Fit Together

We propose Tetris3D, a generative framework for single-image 3D scene reconstruction that recovers objects which are physically and geometrically coherent as a scene. Existing methods often generate objects independently or couple them implicitly, providing limited guidance for ensuring fine-grained spatial compatibili...

Jaeyeon Kim, Jinhyuk Jang, Jongmin Lee et al. · 0 citations
Preprint Sep 2026

FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute

We present FIRE3D, a unified framework that transforms a single RGB image or casual video into interactable 3D scene assets for games and interactive applications in under a minute for up to twelve textured instances including preprocessing. At the core of FIRE3D is a feed-forward inference pipeline that predicts a com...

Hongchi Xia, Tianhang Cheng, Wei-Chiu Ma et al. · 4 citations · ⚡1
Preprint Sep 2026

SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

A framework that combines the merits of both parametric image-to-3D scene models and vision-language models, SceneMosaic is proposed, which obtains the initial candidate from the learned image-based prior, and subsequently evolves the result through VLM agents, ensuring both efficiency and physical validity.

Xingjian Ran · 2 citations
Preprint Aug 2026

NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation

NeoWorld-Pro is proposed, a framework that reformulates monocular scene reconstruction as procedural programming for interactive 3D environments and outperforms open-loop and prior monocular reconstruction methods, while enabling complex downstream tasks such as stable stacking and fine-grained manipulation.

Yu-Meng He, Eachen Soong, Xiao-Tian Yang et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.