Advancing World Simulation With 3D Gaussian Splatting for Large-Scale Environment Reconstruction

Abstract

Advance World Simulation With 3D Gaussian Splatting for Large-Scale Environment Reconstruction is a hands-on lab demonstrating how to reconstruct large-scale outdoor environments using 3D Gaussian Splatting. Additionally, it goes over how deploy these reconstructed scenes for robotic simulation in NVIDIA Isaac Sim. Participants will use the NVIDIA fVDB framework and NVIDIA Omniverse NuRec to transform real-world imagery into interactive, physically simulated scenes. The lab is organized into three parts. In part 1, 3.1, we cover reality capture and data preparation, including Structure-from-Motion, followed by training a 3D Gaussian Splatting model using NuRec’s fVDB pipeline. In part 2, 3.2, we train a model to segment components of the scene, enabling the creation of new assets. Finally, in part 3, 3.3, we export the reconstruction and segments as meshes and USD assets, import them into Isaac Sim, and configure a robot to navigate within the reconstructed environment. Topics include best practices for data capture; SfM and 3DGS in fVDB; scaling to large scenes; optional segmentation and cleanup; mesh and USDZ export; and integration with Isaac Sim for robotics. The lab highlights how NuRec and fVDB support city-scale outdoor reconstruction and how this workflow connects to physical AI and robot testing.