It is proved that the computationally cheaper split space-time attention is equivalent to full space-time attention and is promising to extend VideoSEMA to longer videos with a dilated/sparse temporal attention.
Abstract
We present for video understanding (classification) a split space-time attention model, VideoSEMA, consisting of a scalable and efficient Mamba-like attention (SEMA) block in space and a softmax temporal attention in time. In each frame, SEMA attention applies a local window attention in parallel with a global averaging in a Mamba macro-architecture, which is called Mamba-like. Under certain rank conditions, we prove that the computationally cheaper split space-time attention is equivalent to full space-time attention. On benchmark K400 data sets, VideoSEMA out-performs heavier vision transformer and Mamba models. On benchmark SSv2 data, VideoSEMA leads in top-1 accuracy among models of similar parameter sizes. As image resolution scales up from standard $224^2$ to $1024^2$ on K400 and without fine-tuning, VideoSEMA degrades much more gracefully than VideoMamba in accuracy. It is promising to extend VideoSEMA to longer videos with a dilated/sparse temporal attention.
Recent advances in video understanding have spanned motion, long video, and streaming interaction, driving this field toward real-world applications. Despite this progress, current open-source models remain limited in several ways. They often struggle to generalize across diverse video types, making them effective only in specific domains. High computational demands further restrict their efficiency and scalability. Moreover, most models are only partially open, with key components such as training code, strategy, or datasets unavailable, which hinders reproducibility and slows community-driven development. To address these issues, we introduce VideoChat3, a fully open, efficient, and generalist video-centric MLLM. VideoChat3 advances video understanding through two complementary designs. For efficiency, we introduce Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for Streaming Video Perception, which enables efficient spatiotemporal representation and reduces the cost of processing video inputs during training and inference. For effectiveness, we develop a scalable video data synthesis pipeline that curates three diverse, high-quality training datasets: VideoChat3-Academic2M, VideoChat3-LV116K, and VideoChat3-OL617K, covering general, long-form, and streaming video scenarios, improving the model's generalization across domains. By integrating these designs, VideoChat3 achieves a rare balance of broad generalization and computational efficiency. Experiments across general, long-form, and streaming benchmarks demonstrate that VideoChat3 surpasses prior open-source models with equal or larger parameter counts with only 4B parameters and higher efficiency.
Xinhao Li, Yuhan Zhu, Xiangyun Zeng et al.· 4 citations
In sports AI, human action recognition (HAR) faces a challenge between the expensive Transformer and the one-dimensional state space models (SSMs). Although Transformer has proven success on video tasks, its high computational cost scales quadratically. In contrast, conventional SSMs like Mamba possess linear complexity, but also underperform in the recognition. In this paper, we propose MMViT (Multi-scale Mamba Visual Transformer) with a hierarchical design for improved recognition and objective efficiency. We employ a heterogeneous "Attention-Mamba-Attention" (A-M-A) strategy. It first uses Multi-scale Pooling Attention (MPA) for efficient capture of local spatial feature. As computation-heavy stages come, it transitions to Mamba module with linear complexity to efficiently model long-range temporal context. Finally, attention is re-introduced at latter stages for semantic feature fusion. Also, we introduce a computation-downsampling decoupling (CDD) mechanism to preserve feature coverage during Mamba spatial scaling change. We have validated our approach on SpaceJam and Basketball-51 datasets. Experiments show that MMViT achieves superior performance over strong baselines with substantial margins. Ablation studies show the significance of A-M-A, MPA and CDD. MMViT achieves competitive accuracy among evaluated models and provides a favorable accuracy-efficiency trade-off for video action recognition task.
Jie Dai, Ziyi Quan, Yunxiang Zheng et al.· Scientific Reports· 0 citations
Understanding long-range videos remains a key challenge in computer vision due to high temporal redundancy and computational burden. Despite strong performance of recent models, they are constrained in terms of scalability and generalization when applied to longer video sequences. In this work, we present Keyframe-based Spatio-Temporal Adaptive Representation (K-STAR), a redundancy-aware video summarization framework designed to generate compact and semantically rich representations that are effective in downstream tasks. The proposed method jointly models appearance and motion cues while filtering redundant frames. Importantly, it preserves critical temporal transitions while significantly reducing the number of processed frames. Additionally, each key frame is encoded using object, scene, and background-aware prompts, enabling richer semantic representation. Evaluated on the UCF-101 dataset, K-STAR achieves Top-1 accuracy of 93.06% and Top-5 accuracy of $\mathbf{9 8. 7 3 \%}$, with $\mathbf{5 6} \times$ frame reduction and $\mathbf{1 1. 5} \times$ faster inference, demonstrating competitive performance with substantially improved efficiency.
Rahul Kumar, S. Channappayya· International Conference on...· 0 citations
This work introduces SANA-Video 2.0, a hybrid video diffusion transformer instantiated at 5B and 14B scales under a unified architecture that recovers softmax-level expressiveness at substantially reduced cost, unlocking scalable long, high resolution video generation.
Junsong Chen, Jincheng Yu, Yitong Li et al.· 0 citations
It is concluded that for sparse-frame video grounding, training strategy dominates model scale: a fine-tuned 2B model consistently outperforms a zero-shot 8B model, with or without dense frame access.
Classification of video data is challenging due to the temporally repeated measures and the differential importance of different frames in a video clip. This paper proposes a framework called Frame Importance Voting (FIV), where frame importance weighting and voting are merged as part of a shared inference process to enhance the temporal classification of video data without additional computational burden. Spatial video features were extracted from the video clips using a ResNet-50 architecture, and the temporal relationships were modeled using a two-layer transform encoder. Frame significance was derived adaptively by summing the transformer's attention and confidence scores for each frame, and predictions for categories were made by summing the frame predictions using weighted voting. Results on Kinetics-400 (50 categories, 10,000 clips, 16-32 frames per video) confirmed that FIV achieved 77.4% top-1, 92.5% top-5, and 76.8% top-1, outperforming the aggregators by up to 5.6% (using about 37 million parameters).
Yahya Albugg, M. Alsaleem, O. Hasan· JOIV: International Journal...· 0 citations
