Simulation is a powerful tool for robotics, but building diverse, simulation-ready scenes at scale remains difficult. Reconstructing scenes from single images is a promising alternative, but visual similarity alone is insufficient: objects must also be recovered, accurately placed, and physically grounded. Existing methods that rely on post-hoc physics settling often misplace objects, producing scenes that look correct but behave poorly under manipulation. We present SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. SceneRig combines geometric, visual, and physical tools to reconstruct a faithful 3D scene. An agent then performs edit-and-validate loops inside a physics simulator to achieve physical stability. Across diverse inputs, SceneRig improves instance segmentation, object-centric appearance, and 3D placement while keeping the reconstructed scenes physically stable. Furthermore, SceneRig is effective for downstream robotics: its scenes predict real-robot policy outcomes more accurately than the baseline method and more reliably support replay of recorded manipulations without trajectory adaptation.
Experiments show that Fysiverse-3D-SimReady improves input-view alignment, contact plausibility, and physical stability over existing single-image reconstruction and scene generation baselines, while enabling goal-conditioned physical interactions from a single image.
Lin-Tao Wang, Ming-Yang Sun, Yang Liu et al.· 1 citation
SceneAgent is presented, an agentic pipeline that coordinates these steps through simulator-based visual inspection and repair and combines object segmentation, physical parameter estimation, part decomposition, articulation, and deformable simulation and demonstrates diverse simulated interactions and preliminary poli...
These readouts suggest that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images, suggesting that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images.
Xi-Rui Li, Peng Shi, Ming-Wen Dong et al.· 0 citations
A framework that combines the merits of both parametric image-to-3D scene models and vision-language models, SceneMosaic is proposed, which obtains the initial candidate from the learned image-based prior, and subsequently evolves the result through VLM agents, ensuring both efficiency and physical validity.
Results show that improving the quality of the environment reconstruction through higher visual fidelity, authored physics and metric scale makes the simulation more faithful to the real world and narrows the sim-to-real gap.
CoDimRecon is presented, an agentic framework that reconstructs editable scenes containing rigid, articulated, and deformable objects from multi-view RGB observations that achieves competitive compositional reconstruction accuracy while additionally producing deformable assets for rod, shell, and solid simulation.
Shu-Zhao Xie, Le-Lin Wang, Gu-Ying Lin et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduJul 30, 2026
From feet to fingertips — we are teaching robots intelligent whole-body control, fine dexterity, and teamwork to complete a broad range of complex tasks.
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
