Vision Mamba With Joint Spatiotemporal Features for Efficient Video Representation Learning in Self-Supervised Scheme

Abstract

Video Anomaly Detection (VAD) faces persistent challenges, including annotated data availability, contextual dependency, and elevated false alarm rates. While Self-Supervised Learning (SSL) effectively mitigates label scarcity, the current Self-Supervised Multi-Task Learning (SSMTL) framework encounters two primary challenges. The first challenge pertains to the susceptibility of Conv3D-based encoders to overfitting. This study investigated several overfitting mitigation strategies integrated into the encoder architecture to address this issue. The second challenge concerns the quadratic computational costs inherent in Transformer-based encoders. As a solution, this study introduces Vision Mamba (ViM), leveraging a Selective State Space Model to capture long-range temporal dependencies with linear computational complexity. This efficiency is theoretically substantiated by an asymptotic time complexity analysis, demonstrating ViM’s superiority over Vision Transformers (ViT) regarding data depth dimensions and architectural depth layers. The comprehensive experiments yielded two key findings. First, the empirical results on UCSD Ped2 demonstrate that a 0.3 dropout rate provides superior stability for mitigating overfitting in Conv3D baselines compared to standard regularization. Second, evaluations on a theft-focused UCF-Crime subset confirm ViM as the most lightweight architecture, reducing Floating Point Operations (FLOPs) by a factor of 2-4 relative to alternatives. In terms of performance, ViM outperformed the Conv3D baseline (avg. +0.012) and rival VideoSwin (avg. -0.017), although it trails ViT in the AUC-ROC, Precision, and Recall metrics. Finally, this study validates the ViM as a highly efficient solution that balances computational feasibility with good performance in detecting complex criminal activities at the frame level.