2025· Neural Information Processing Systems· pp. 61089-61107· 1 citation· 50 references
Computer Science
TL;DR
This novel E-VFI framework diverges from approaches reliant on direct image-level supervision by constructing multilevel, degradation-insensitive semantic perceptual supervisory signals to enhance the perceptual realism and multi-scene generalization of the model's predictions.
Abstract
Event cameras, with their capacity to provide high temporal resolution information between frames, are increasingly utilized for video frame interpolation (VFI) in challenging scenarios characterized by high-speed motion and significant occlusion. However, prevalent issues of blur and distortion within the keyframes and ground truth data used for training and inference in these demanding conditions are frequently overlooked. This oversight impedes the perceptual realism and multi-scene generalization capabilities of existing event-based VFI (E-VFI) methods when generating interpolated frames. Motivated by the observation that semantic-perceptual discrepancies between degraded and pristine images are considerably smaller than their image-level differences, we introduce EPA. This novel E-VFI framework diverges from approaches reliant on direct image-level supervision by constructing multilevel, degradation-insensitive semantic perceptual supervisory signals to enhance the perceptual realism and multi-scene generalization of the model’s predictions. Specifically, EPA operates in two phases: it first employs a DINO-based perceptual extractor, a customized style adapter, and a reconstruction generator to derive multi-layered, degradation-insensitive semantic-perceptual features ( S ). Second, a novel Bidirectional Event-Guided Alignment (BEGA) module utilizes deformable convolutions to align perceptual features from keyframes to ground truth with inter-frame temporal guidance extracted from event signals. By decoupling the learning process from direct image-level supervision, EPA enhances model robustness against degraded keyframes and unreliable ground truth information. Extensive experiments demonstrate that this approach yields interpolated frames more consistent with human perceptual preferences. Codes are available at https://github.com/yuhan0802/EPA.
This work proposes an adapter-based framework that incorporates event-derived cues into a pre-trained image-to-video diffusion model with minimal architectural changes and consistently outperforms existing state-of-the-art approaches.
Guixu Lin, Yuyang Yu, Xiang Ji et al.· 0 citations
Despite the success of diffusion models in Video Frame Interpolation (VFI), existing methods still suffer from two critical limitations. First, latent diffusion inevitably loses fine-grained details when reconstructing images from latent representations back to the pixel space. Second, multi-step sampling incurs prohibitive memory consumption and inference latency. To address these issues, we propose SPEED, a one-step pixel diffusion framework for high-quality VFI. Specifically, SPEED employs a progressive multi-stage architecture with dynamic patch scaling to effectively learn multi-scale motion, structural, and appearance representations. Furthermore, we propose a novel Noise-Update-Only Attention mechanism to prevent semantic degradation of the clean condition frames while reducing the computational overhead by nearly 50%. Besides, we introduce a Drift-aware Timestep Sampling strategy coupled with a tailored training objective to directly predict images in the pixel space, enabling one-step inference without compromising the quality of the generated frames. Extensive experiments show that SPEED achieves state-of-the-art performance. On SNU-FILM, SPEED reduces LPIPS by 8.8% while delivering 63.3% faster inference and 10.6% lower memory usage. On challenging 4K benchmarks, it further surpasses prior methods by up to 51.5% in LPIPS.
Zihao Zhang, Haoyu Zhao, Siqian Yang et al.· 0 citations
High-Compression videos suffer from severe distortions, among which degradation in person regions has the greatest impact on viewers’ immersive experience. Existing quality enhancement techniques usually focus on overall image denoising or super-resolution, often overlooking the crucial recovery of fine structures in these essential person regions. To address these challenges, the research introduces a novel framework titled Person Region Restoration Driven by Perceptual Fidelity (PRRDPF), which combines long-range dependency features with perceptual structure loss for enhanced generative restoration. Specifically, first, the research constructs a high-fidelity distorted person-region dataset via a closed-loop degradation pipeline, addressing the lack of paired datasets. Secondly, a Temporal Gated Fusion (TGF) block is designed to use gated convolutions for selectively recovering high-frequency features while capturing local and global dependencies. Finally, a Structural Similarity Index Measure (SSIM)-based dynamic weighted adversarial loss is proposed to prioritize the restoration of visual texture details. Experimental results validate that PRRDPF significantly outperforms the best models in Peak Signal-to-Noise Ratio (PSNR), SSIM, and Learned Perceptual Image Patch Similarity (LPIPS), effectively mitigating artifacts and enhancing clarity in person visuals. This framework presents a promising approach for intelligent video coding integrated with generative artificial intelligence and holds significant potential for practical applications.
Linyun Liu, Li Yu, Jiaxin Zeng et al.· IEEE Signal Processing Lette...· 0 citations
It is concluded that for sparse-frame video grounding, training strategy dominates model scale: a fine-tuned 2B model consistently outperforms a zero-shot 8B model, with or without dense frame access.
Deep learning-driven representations such as neural radiance fields (NeRFs) and 3D Gaussian splatting (3DGS) have revolutionized the field of dynamic 3D scene reconstruction with improved visual precision and scalability. However, the reconstruction of fast-moving objects remains a challenge; existing methods based on conventional frame-based videos often struggle in scenarios such as sports events and animal videography. We propose an event-RGB fusion Gaussian splatting (ERF-GS) framework that integrates event information into both optimization and densification stages of the Gaussian splatting pipeline, taking advantage of novel event sensors with high frame-rate. Unlike many other event-assisted scene reconstruction methods, ERF-GS was developed using realistic simulation settings and realizes event-based learning detached from RGB inputs. This design enables its application beyond straightforward synthetic data into the realm of natural video with complex layout, low frame rates and severe motion blur. Our experiments show that ERF-GS outperforms both the 4DGS baseline and the concurrent E-D3DGS on different variants of the Neu3D and Nvidia datasets which include blurry RGB frames and disjoint RGB-event viewpoints. Our code is available at https://github.com/andrewbxy/ERF-GS.
Xiaoyang Bai, Zhenyang Li, Weiwei Xu et al.· 0 citations
Understanding long-range videos remains a key challenge in computer vision due to high temporal redundancy and computational burden. Despite strong performance of recent models, they are constrained in terms of scalability and generalization when applied to longer video sequences. In this work, we present Keyframe-based Spatio-Temporal Adaptive Representation (K-STAR), a redundancy-aware video summarization framework designed to generate compact and semantically rich representations that are effective in downstream tasks. The proposed method jointly models appearance and motion cues while filtering redundant frames. Importantly, it preserves critical temporal transitions while significantly reducing the number of processed frames. Additionally, each key frame is encoded using object, scene, and background-aware prompts, enabling richer semantic representation. Evaluated on the UCF-101 dataset, K-STAR achieves Top-1 accuracy of 93.06% and Top-5 accuracy of $\mathbf{9 8. 7 3 \%}$, with $\mathbf{5 6} \times$ frame reduction and $\mathbf{1 1. 5} \times$ faster inference, demonstrating competitive performance with substantially improved efficiency.
Rahul Kumar, S. Channappayya· International Conference on...· 0 citations