MotionStruct4D: Discovering Motion Structure of Gaussian Splatting for Video-to-4D Generation

Abstract

Monocular Video-to-4D generation faces the fundamental challenge of inferring plausible 3D geometry and motion from limited single-view inputs. We present MotionStruct4D, a novel approach that discovers and exploits the underlying motion structure of 3D Gaussian Splatting for high-quality video-to-4D generation. Our key insight is that real-world motion can be effectively decomposed into coarse rigid transformations that capture principal movements, complemented by detailed non-rigid deformations that account for fine-grained details. MotionStruct4D introduces: (1) a self-supervised motion structure discovery module that identifies quasi-rigid parts by preserving spatiotemporal relationships without explicit 3D supervision, and (2) a weighted dense-to-sparse optimization architecture that transitions from dense per-Gaussian deformation to sparse control points, effectively integrating rigid and non-rigid motion components through adaptive weighted fusion. This design addresses the parametric imbalance between rigid and non-rigid motions and effectively models 3D movements across different kinematic patterns. To evaluate our approach on challenging scenarios, we curate a comprehensive benchmark dataset featuring substantial object displacement and diverse articulated motion patterns. Experimental results demonstrate MotionStruct4D’s superior performance in motion fidelity and novel viewpoint synthesis quality, while also providing interpretable motion structure decomposition that reveals meaningful quasi-rigid part segmentation.