TL;DR
This paper presents an innovative method that leverages user-specified action paths to guide the 4D scene generation that dynamically synchronizes motions in the action path domain with their corresponding contents in the time domain.
Novel view rendering of large and complex reconstructed scenes is becoming increasingly photorealistic. However, most reconstructions remain static and lack the ambient motion that makes environments immersive. We present AniGS, a method for scene-level animation of 3D Gaussian Splatting (3DGS) reconstructions that adds subtle, distributed dynamics, e.g., vegetation motion, while preserving rigid structures. Unlike existing 3D animation techniques which are limited to object-centric subjects or small regions, AniGS is designed for large, cluttered, navigable scenes. AniGS represents the scene with a canonical 3DGS and models motion using a time-conditioned deformation field. To animate the entire scene, we leverage a pretrained video diffusion model and introduce an iterative dataset--model update strategy that progressively expands viewpoint coverage and repeatedly updates camera-fixed training videos using a render-and-refine scheme. To prevent artifacts from unintended motion in static areas, we further introduce a composed video-to-video refinement scheme that restricts motion to desired regions. Experiments on five real-world, large-scale outdoor scenes demonstrate that AniGS produces natural ambient dynamics and high-quality novel view videos, enabling more immersive viewing experiences of reconstructed environments.
Autonomous robots typically need to construct representations of their surroundings and adapt their motions to the geometry of their environment. Here, we tackle the problem of constructing a policy model for collision-free motion generation, consistent with the environment, from a single input RGB image. Extracting 3D structures from a single image often involves monocular depth estimation. De-velopments in depth estimation have given rise to large pre-trained models such as DepthAnything . However, using outputs of these models for downstream motion generation is challenging due to frustum-shaped errors that arise. Instead, we propose a framework known as Video-Generation Environment Representation (VGER), which leverages the advances of large-scale video generation models to generate a moving camera video conditioned on the input image. Frames of this video, which form a multiview dataset, are then input into a pre-trained 3D foundation model to produce a dense point cloud. We then introduce a multi-scale noise approach to train an implicit representation of the environment structure and build a motion generation model that complies with the geometry of the representation. We extensively evaluate VGER over a diverse set of indoor and outdoor environments. We demonstrate its ability to produce smooth motions that account for the captured geometry of a scene, all from a single RGB input image.
This paper tackles the challenge of novel view synthesis in complex scenes with under-constrained motion, as captured in monocular videos. Existing methods mainly focus on handling motion restricted within a bounded 3D volume, relying on spatio-temporal information to drive dynamic Gaussian deformations. However, due to the inherent motion ambiguities in monocular dynamic 3D representations and the limited observations, these methods face challenges in handling such scenes, often leading to incomplete geometry and boundary artifacts. To mitigate these issues, we propose a semantics-guided scene decoupling module that separates Gaussian primitives into static and dynamic components based on motion vectors. Further more, to enhance the capability in modeling non-rigid motions, we introduce a motion-aware densification module for motion compensation, which alleviates the incomplete rendering of dynamic objects caused by insufficient spatio-temporal information. Experimental results on real-world datasets demonstrate that our approach outperforms state-of-the-art methods in preserving both the integrity and detailed appearance of moving objects in dynamic scenes.
This work revisits the role of positional encoding in video diffusion transformers and shows that it provides a useful spatial bias for geometry-aware control, and introduces a geometry-aware cross-attention mechanism that enables target video latent tokens to attend to structured context tokens derived from reference images or frames.
This work shows that camera motion, object trajectories, and depth can be unified into a single 3D point-track representation, from which one model performs joint camera and object control, depth editing, and motion transfer in a single forward pass, enabling interactive 4D-controllable streaming generation for the first time.