Struct-GStream is proposed, which can achieve efficient FVV streaming using structured 3D Gaussians (3DGs) and introduces dynamic anchor points to generate structured 3DGs to construct basic scenes and model approximate scene movements based on the assumption of local rigidity in object motion.
Abstract
Constructing photorealistic Free-Viewpoint Videos (FVVs) of dynamic scenes from a set of posed 2D images has been an intriguing yet challenging task in computer vision. Methods based on neural rendering achieve high-fidelity image quality in FVV construction. However, most of these methods are unable to achieve real-time rendering and often require complete video sequences to train. Despite the existence of some online training methods capable of rendering FVVs in real time, they struggle to meet the requirements for storage and training time for downstream applications. To overcome this problem, we propose Struct-GStream, which can achieve efficient FVV streaming using structured 3D Gaussians (3DGs). Specifically, we introduce dynamic anchor points to generate structured 3DGs to construct basic scenes and model approximate scene movements based on the assumption of local rigidity in object motion. Besides, we introduce a global free 3DGs patching strategy involving free 3DGs'generation, pruning, and optimization to patch and model deficient areas and emerging objects. Our method achieves fast training at low bitrates while maintaining high rendering quality. Extensive experiments demonstrate that Struct-GStream significantly outperforms existing online training methods for FVV construction in terms of training time, storage, and rendering quality while maintaining competitive rendering speed.
We present LiveLight, the first diffusion-based framework for real-time streaming video relighting with interactive 3D lighting control. Achieving this is non-trivial, as it requires overcoming three critical challenges: effectively injecting dynamic 3D lighting into a diffusion model, maintaining high-fidelity generation under an extremely low NFE (Number of Function Evaluations) budget for real-time speed, and facilitating continuous streaming for interactive control. To address these pain points, we propose three key designs. First, for accurate lighting injection, we propose a lightweight adapter that feeds Multi-Plane Light Irradiance (MPLI) conditions-depth-aware irradiance maps encoding 3D lighting geometry-directly into the diffusion backbone. Second, to prevent rendering quality degradation at low NFEs towards real-time distillation, we introduce a geometry-guided feedback branch. This training-time constraint leverages a frozen geometry estimator to enforce depth- and normal-consistent relighting, ensuring geometrically plausible shading without adding inference overhead. Finally, to enable streaming interaction, we develop a progressive rolling-window strategy that maintains a denoising ladder of latent chunks at varying noise levels. By propagating intermediate states, this strategy guarantees temporal coherence and supports arbitrarily long video relighting with per-frame reference refresh. Extensive experiments on real-world and synthetic benchmarks demonstrate that LiveLight achieves state-of-the-art relighting quality while running at real-time speed, significantly outperforming offline baselines in temporal stability, lighting controllability, and user preference. To foster real-time interactive relighting research, we will publicly release our models, training data, and synthetic data generator.
Yue Ma, Jiangming Wang, Yucheng Wang et al.· 1 citation
This paper tackles the challenge of novel view synthesis in complex scenes with under-constrained motion, as captured in monocular videos. Existing methods mainly focus on handling motion restricted within a bounded 3D volume, relying on spatio-temporal information to drive dynamic Gaussian deformations. However, due to the inherent motion ambiguities in monocular dynamic 3D representations and the limited observations, these methods face challenges in handling such scenes, often leading to incomplete geometry and boundary artifacts. To mitigate these issues, we propose a semantics-guided scene decoupling module that separates Gaussian primitives into static and dynamic components based on motion vectors. Further more, to enhance the capability in modeling non-rigid motions, we introduce a motion-aware densification module for motion compensation, which alleviates the incomplete rendering of dynamic objects caused by insufficient spatio-temporal information. Experimental results on real-world datasets demonstrate that our approach outperforms state-of-the-art methods in preserving both the integrity and detailed appearance of moving objects in dynamic scenes.
Chulin Zhao, Xue Wang, Guoqing Zhou et al.· IEEE Transactions on Visuali...· 0 citations
Real-time visualization of massive 3D virtual environments, such as city-scale digital twins and immersive simulations, remains a core challenge in computer graphics. While 3D Gaussian Splatting (3DGS) achieves photorealistic rendering at interactive rates, its explicit Gaussian representation incurs substantial GPU memory overhead, limiting deployment on resource-constrained devices. Existing 3DGS Level-of-Detail (LoD) methods are often data-dependent, requiring access to large collections of original training images, and they rarely adapt to dynamic hardware budgets, leading to unstable performance on low-VRAM GPUs. We present InfoLoD, a data-free and hardware-aware framework for hierarchical LoD construction and large-scale 3DGS rendering. InfoLoD introduces a Fisher-guided self-distillation scheme that uses the Fisher Information Matrix (FIM) to select geometrically valid, information-rich pseudo viewpoints, enabling LoD training directly from a pre-trained 3DGS model without any original images. To operate under strict VRAM constraints, we further propose a budget-aware out-of-core streaming system with spatio-temporal working-set management and resource-adaptive LoD scheduling to balance fidelity and throughput. Experiments on challenging outdoor scenes show that InfoLoD enables stable real-time rendering on resource-constrained GPUs, delivering higher efficiency than prior baselines while maintaining competitive visual quality.
Zhenyu Xia, Pengcheng Han, Lin Chen et al.· IEEE Transactions on Visuali...· 0 citations
3D scene representation techniques such as neural radiance fields (NeRFs) and Gaussian splatting have made substantial progress in novel view synthesis, achieving high-quality renderings from arbitrary view angles. More recently, such techniques have been extended to dynamic 3D scenes; however, achieving sustainable online free-viewpoint video (FVV) streaming remains challenging, especially for longer videos, due to significant storage demands of detailed scene representations and high reconstruction/rendering speed needs. To address these challenges, we propose Quantized Anchored Residual Coding Gaussian Streaming (QuARC-GS), a quantization-aware 4D scene optimization framework for online dynamic scene reconstruction that achieves ultra-high compression while maintaining reconstruction speed and quality. QuARC-GS represents a scene using a single canonical frame and highly compressed per-frame residuals. Specifically, we compress each residual through two complementary strategies targeting motion, appearance, and densification. We introduce quantization-aware anchor deformation, which suppresses insignificant motion updates while preserving meaningful deformations, maintaining reconstruction quality under low-storage streaming. Furthermore, we design a change-gated densification strategy that allocates new Gaussians only in regions exhibiting genuine temporal changes, effectively eliminating redundant appearance updates and reducing storage overhead. Extensive experiments on widely used datasets demonstrate that QuARC-GS enables competitive reconstruction quality and training speed while cutting per-frame storage by up to 11$\times$ compared to the state-of-the-art.
V. Nguyen, Yuchen Wang, Kyung Chul Lee et al.· 0 citations
We present an empirical study of free-view video compression for dynamic scenes reconstructed with 3D Gaussian Splatting, examining how practical pipeline design choices affect reconstruction fidelity and storage efficiency. Rather than introducing new representations, we analyze how commonly used components, including temporal chunking, deformation-based reconstruction, and quantization-aware training, interact in practice. We observe that partitioning long sequences into shorter temporal segments, such as GOPs, simplifies optimization and improves reconstruction fidelity, but can introduce additional storage overhead. We further show that encouraging smooth motion vectors across both space and time produces deformation signals that are easier for standard video codecs to compress, leading to improved rate–distortion performance. When integrated into a unified pipeline, these design choices consistently benefit different deformation-based reconstruction methods. Across multiple datasets, our approach achieves 20% storage reduction compared with state-of-the-art methods while preserving or improving visual quality, and we discuss sources of variability and ambiguity in current training and evaluation protocols.
Mingyang Song, Yang Zhang, Siyu Tang et al.· International Conference on...· 0 citations
Recent advances in 3D Gaussian Splatting (3DGS) have enabled feed-forward, on-the-fly reconstruction of sequential input views. However, existing methods often predict per-pixel Gaussians and combine Gaussians from all views as the scene representation, leading to substantial redundancies and geometric inconsistencies in long-duration video sequences. To address this, we propose SaLon3R, a novel framework for Structure-aware, Long-term 3DGS Reconstruction. Our method eliminates redundancy by introducing compact anchor primitives as a replacement for per-pixel Gaussians. These primitives are derived through a differentiable, saliency-aware Gaussian quantization process designed to preserve fidelity while ensuring a compact representation. Specifically, a foundational 3D reconstruction model is employed to predict a saliency map encoding regional geometric complexity. Guided by this saliency map, we compress redundant Gaussian primitives into compact anchors by prioritizing high-complexity regions. Furthermore, we introduce a 3D Point Transformer to overcome geometric inconsistencies caused by long-term accumulative errors. It refines attributes and saliency of the anchor primitives leveraging the learned spatial structural priors in 3D space. Without known camera parameters or test-time optimization, our approach effectively prunes the redundant 3DGS and resolves artifacts in a single feed-forward pass. Experiments on multiple datasets demonstrate our approach outperform state-of-the-arts on both novel view synthesis and depth estimation, while exhibiting superior efficiency (>\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$>\!$$\end{document}10FPS), robustness, and generalization ability for long-term feed-forward Gaussian reconstruction.
Jiaxin Guo, Tongfan Guan, Wenzhen Dong et al.· International Journal of Com...· 5 citations· ⚡1