Skip to content

VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis

Sep 2026 · arXiv.org · Vol abs/2609.33253 · 0 citations · 68 references
Computer Science

TL;DR

This work presents VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis, and introduces robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness.

Abstract

We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-{\Omega} into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.

View source

Similar papers

Preprint Sep 2026

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

GeoVerse is a framework that synthesizes world-consistent novel views by performing generation within the geometric latent space of a pretrained 3D foundation model and injecting appearance priors from a video generative model via a ControlNet-style adapter.

Ke-Rui Ren, Tao Lu, Lin-Ning Xu et al. · 0 citations
#diffusion models Preprint Sep 2026

StreetDiff: Multi-view Street Scenes Generation via Cross-view Consistent Multi-view Stable Diffusion with Structure Prompts

The proposed StreetDiff introduces a Panorama--Perspective Synergy design to decouple global layout reasoning from local detail synthesis, and incorporates a Panorama Alignment Module (PAM) that establishes spherical-projection-based attention constraints across views to achieve robust cross-view coherence in challengi...

Qi Zhang, Yanyifan Wang, Wei-Yuan Zhang et al. · 0 citations
#artificial intelligence Preprint Aug 2026

Video Generative Models as Geometry Learner

This work repurposes pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task, and inherits naturally structured knowledge and richer priors from the video model, enabling more data efficient and effective learning of...

Haosen Yang, Ji-Fei Song, Zhensong Zhang et al. · 1 citation
Sep 2026

D3GS: Depth, DINO, and RGB Diffusion Co-Guided 3D Gaussian Splatting for Sparse-View Reconstruction

Novel view synthesis from sparse inputs remains challenging for 3D Gaussian Splatting (3DGS) due to ambiguous geometry, cross-view inconsistency, and missing details in under-constrained regions, resulting in degraded reconstruction and unstable rendering. To tackle these issues, we propose D$^{3}$GS, a Depth-DINO-Diff...

Yun-Qi Gao, Zhan-Feng Liao, Han-Zhang Tu et al. · 0 citations
Preprint Aug 2026

Visual Geometry Foundation-Aware Gaussians for Single-Frame Surround-View Driving Reconstruction

VGGD, a visual geometry foundation-aware 3D Gaussian Splatting framework for feed-forward surround-view driving reconstruction, which shifts geometric modeling to the frontend and adapts foundation priors to the driving camera setting and achieves the best overall rendering quality among the compared methods and improv...

Junhong Lin, Jinlong Wang, Xianda Guo et al. · 0 citations
Open access 2026

SSGLD: Spatial-Semantic Guided Latent Diffusion for Person Image Synthesis

Results show that SSGLD effectively resolves the trade-off between texture fidelity and pose-variation robustness in high-resolution paired PGPIS under sparse 2D skeleton conditioning.

Yi-Heng Xu, Zhen-Yin Zhang, Geng-Shen Chen · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.