Mar 2024· New generation computing· Vol 44· 13 citations· 69 references
Computer Science
TL;DR
The ablation study shows that when using SSMs for temporal modeling, incorporating bidirectionality and selective scans enhances video generation performance, and SSM-based models incur lower computational cost to achieve the same Fréchet Video Distance as attention-based models.
Abstract
Given the remarkable achievements in image generation using diffusion models, the research community has shown increasing interest in extending these models to video generation. Recent diffusion models for video generation have predominantly utilized attention layers to extract temporal features. However, attention layers are limited by their computational cost, which increases quadratically with sequence length. This limitation poses significant challenges when generating longer video sequences using diffusion models. To overcome these challenges, we propose to leverage state-space models (SSMs) as temporal feature extractors. SSMs (e.g., Mamba) have recently garnered attention as promising alternatives owing to their linear-time memory and time consumption relative to the sequence length. Employing SSMs to capture temporal dependencies in video generation enables significantly higher generative performance at the same computational cost (e.g., memory usage, inference time) compared to attention-based methods, particularly for long-term sequences. For various model sizes, we comprehensively evaluated multiple long-term video datasets: MineRL Navigate, GQN-Mazes, and CARLA-Town01. For 256-frame video sequences, SSM-based models incur lower computational cost to achieve the same Fréchet Video Distance as attention-based models. Furthermore, the ablation study shows that when using SSMs for temporal modeling, incorporating bidirectionality and selective scans enhances video generation performance. Our code is available at https://anonymous.4open.science/r/SSM-Meets-Video-Diffusion-Models-067D/README.md.
MobileWan becomes the first 5B-scale video diffusion model deployable on a commercial mobile device and proposes a learnable attention head pruning method based on binary per-head gates optimized end-to-end using a noise-biased sparsity objective and distillation-based finetuning.
Mohsen Ghafoorian, Denis Korzhenkov, Adil Karjauv et al.· 1 citation
Recently, diffusion models have made great progress in video generation. However, most existing video diffusion models are trained with short videos, and degrade when extrapolated to long videos, struggling to maintain long-range temporal coherence while retaining diverse motions. To generate consistent, high-quality and dynamic long videos, we propose Diff-VF, a training-free, plug-and-play and model-agnostic framework that converts existing short-video diffusion backbones into long-video generators without modifying or fine-tuning the base model. Diff-VF couples three complementary strategies: Hybrid Noise Initialization (HNI) to constrain global semantics, Weighted Window Sampling (WWS) to remove inter-window discontinuities, and Temporal Extended Sampling (TES) to establish long-range dependencies with a timestep-varying fusion. We further extend Diff-VF to long-video enhancement via Skip Residual Guidance that balances fidelity and realism through timestep-dependent guidance. VBench-Long evaluation results show that Diff-VF achieves a more favorable balance between temporal coherence and motion diversity than base models and recent training-free long video generation baselines, including FreeNoise, FreeLong, and RIFLEx, while maintaining competitive frame-wise quality. Experiments on two base models demonstrate the applicability to video diffusion models with different spatial-temporal modeling strategies. Extensive ablations validate the contribution of each component and hyperparameters.
Haoning Yang, Xinyuan Chen, Yaohui Wang et al.· ACM Transactions on Multimed...· 1 citation
Gen4U (Generation for Understanding), a framework that repurposes these generative representations with a single forward pass, is introduced, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.
Michael King, Aravindh Mahendran, M. Grimes et al.· 0 citations
A novel framework that integrates temporally consistent diffusion models with dynamic scene-graph guidance that structurally constrains the generative process, ensuring that objects, their attributes, and their interrelationships remain stable over extended durations is introduced.
Jacob A. Jenkins· Journal of innovative resear...· 0 citations
This work proposes Ms.Forcing, an efficient streaming video generation paradigm that adapts spatial granularity to each state's noise level and introduces Homogeneous-Noise-Level DMD, which assembles each fake video from clean predictions sharing the same source noise level, thereby reducing the mismatch between DMD training sequences and inference-time rollouts.
Zekun Li, Xiaoyan Cong, Hongyu Li et al.· 0 citations
Latent diffusion models have emerged as a dominant framework for high-fidelity image and video synthesis, operating in compact latent spaces with variational autoencoders (VAEs) to enhance computational efficiency without compromising visual quality. However, conventional VAEs are suboptimal for video data as they employ fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector which is jointly learned with latent tokens. By evaluating each token's content-richness as keep-or-drop probability, the token selector effectively discards uninformative tokens, naturally allowing data-dependent compression. Applying adaptive tokenization to diffusion models may cause spatial misalignment, as token dropping can disturb the original spatio-temporal structure. To alleviate this issue, we propose two position-prediction strategies: cascaded and joint generation, to ensure spatial consistency. We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis on video data reveals that this improvement is primarily achieved by reducing spatio-temporal redundancy and removing uninformative tokens, as supported by both quantitative and qualitative results.
Yeonkyeong Lee, Hyun-Young Go, Jongmin Kim et al.· 0 citations
