The method sets a new state of the art in 4D reconstruction and seamlessly generalizes to in-the-wild videos with large viewpoint changes and dynamic motions, improving both novel-view synthesis and the underlying 3D motion.
Abstract
We present World from Motion, a method for generating freely renderable dynamic 3D Gaussian representations from monocular videos. Our approach conditions a video model on dense, pixel-aligned renderings that encode appearance, geometry, and 3D scene motion along both input and target camera trajectories to correct rendering artifacts and fill in missing regions from an initial reconstruction. To train this model, we construct a dataset of aligned multiview video pairs and dynamic 3DGS representations, with simulated artifacts characteristic of monocular reconstruction. At test time, we distill the model's generations, including newly observed regions and motions, back into a single consistent, high-quality dynamic 3DGS, improving both novel-view synthesis and the underlying 3D motion. Our method sets a new state of the art in 4D reconstruction and seamlessly generalizes to in-the-wild videos with large viewpoint changes and dynamic motions.
This paper tackles the challenge of novel view synthesis in complex scenes with under-constrained motion, as captured in monocular videos. Existing methods mainly focus on handling motion restricted within a bounded 3D volume, relying on spatio-temporal information to drive dynamic Gaussian deformations. However, due to the inherent motion ambiguities in monocular dynamic 3D representations and the limited observations, these methods face challenges in handling such scenes, often leading to incomplete geometry and boundary artifacts. To mitigate these issues, we propose a semantics-guided scene decoupling module that separates Gaussian primitives into static and dynamic components based on motion vectors. Further more, to enhance the capability in modeling non-rigid motions, we introduce a motion-aware densification module for motion compensation, which alleviates the incomplete rendering of dynamic objects caused by insufficient spatio-temporal information. Experimental results on real-world datasets demonstrate that our approach outperforms state-of-the-art methods in preserving both the integrity and detailed appearance of moving objects in dynamic scenes.
Chulin Zhao, Xue Wang, Guoqing Zhou et al.· IEEE Transactions on Visuali...· 0 citations
Novel view rendering of large and complex reconstructed scenes is becoming increasingly photorealistic. However, most reconstructions remain static and lack the ambient motion that makes environments immersive. We present AniGS, a method for scene-level animation of 3D Gaussian Splatting (3DGS) reconstructions that adds subtle, distributed dynamics, e.g., vegetation motion, while preserving rigid structures. Unlike existing 3D animation techniques which are limited to object-centric subjects or small regions, AniGS is designed for large, cluttered, navigable scenes. AniGS represents the scene with a canonical 3DGS and models motion using a time-conditioned deformation field. To animate the entire scene, we leverage a pretrained video diffusion model and introduce an iterative dataset--model update strategy that progressively expands viewpoint coverage and repeatedly updates camera-fixed training videos using a render-and-refine scheme. To prevent artifacts from unintended motion in static areas, we further introduce a composed video-to-video refinement scheme that restricts motion to desired regions. Experiments on five real-world, large-scale outdoor scenes demonstrate that AniGS produces natural ambient dynamics and high-quality novel view videos, enabling more immersive viewing experiences of reconstructed environments.
Yen-Chi Cheng, Chen Gao, Chuhan Chen et al.· 0 citations
OmniX achieves state-of-the-art performance on dense 3D point trajectory prediction and 3D point tracking, while also demonstrating competitive results on video depth estimation and camera pose estimation.
Yanqin Jiang, Tengfei Wang, Zhengwei Wang et al.· 2 citations
Autonomous robots typically need to construct representations of their surroundings and adapt their motions to the geometry of their environment. Here, we tackle the problem of constructing a policy model for collision-free motion generation, consistent with the environment, from a single input RGB image. Extracting 3D structures from a single image often involves monocular depth estimation. De-velopments in depth estimation have given rise to large pre-trained models such as DepthAnything . However, using outputs of these models for downstream motion generation is challenging due to frustum-shaped errors that arise. Instead, we propose a framework known as Video-Generation Environment Representation (VGER), which leverages the advances of large-scale video generation models to generate a moving camera video conditioned on the input image. Frames of this video, which form a multiview dataset, are then input into a pre-trained 3D foundation model to produce a dense point cloud. We then introduce a multi-scale noise approach to train an implicit representation of the environment structure and build a motion generation model that complies with the geometry of the representation. We extensively evaluate VGER over a diverse set of indoor and outdoor environments. We demonstrate its ability to produce smooth motions that account for the captured geometry of a scene, all from a single RGB input image.
Weiming Zhi, Ziyong Ma, Tianyi Zhang et al.· Neural Information Processin...· 0 citations
D, a reference-guided renderer that extends Wan2.2 camera control from Plucker rays alone to a joint camera-plus-geometry interface and projects a neural 4D G-buffer from the animated mesh and injects it through a widened control adapter while preserving the pretrained image-to-video prior, supporting tracking+world-position correspondence as a practical 4D rendering condition.
Junhao Chen, Mingjin Chen, He Zhang et al.· 0 citations
This work shows that camera motion, object trajectories, and depth can be unified into a single 3D point-track representation, from which one model performs joint camera and object control, depth editing, and motion transfer in a single forward pass, enabling interactive 4D-controllable streaming generation for the first time.
Shiqian Li, Chenguo Lin, Zhi-Guang Liu et al.· 0 citations