VistaDreamer: Zero-shot Single-view Scene Reconstruction by Multi-view Consistency Sampling.

Abstract

In this paper, we propose VistaDreamer, a novel framework to reconstruct a 3D scene from a single-view image. Recent diffusion models enable generating high-quality novel-view images from a single-view input image. However, maintaining consistency across generated novel views remains challenging, often leading to noisy or distorted 3D reconstructions. VistaDreamer addresses this problem by a two-stage pipeline. In the first stage, VistaDreamer builds a global coarse 3D scaffold by zooming out a little step with inpainted boundaries and an estimated depth map. Then, on this global scaffold, iterative RGB-D inpainting generates novel-view images to fill the holes, guided by several geometric consistency constraints. In the second stage, we further enhance the consistency between the generated novel-view images by a novel training-free Multi-view Consistency Sampling (MCS) that introduces multi-view consistency constraints in reverse sampling process of diffusion models. VistaDreamer framework is compatible with various geometric estimators, diffusion models and samplers. Extensive experiments show that, without task-specific training or fine-tuning, it achieves good reconstruction quality and strong generalization across diverse scenes and trajectories, with ablation studies confirming the effectiveness and trends of our designs.