Hierarchical Multimodal Video Intelligence: A Unified Framework for Adaptive Surveillance, Anomaly Detection and Continual Learning

Abstract

Modern surveillance systems must both detect anomalies accurately and continuously adapt to changing operational environments. In this paper implemented a hierarchical multimodal video intelligence framework that combines multiple different forms of understanding or pattern recognition such as motion analysis, object detection, vision language reasoning and human feedback into one adaptive surveillance system. The framework uses YOLOv8n person detection, farneback optical flow motion classification, and Google Gemma 3n as a vision language model for hierarchical scene understanding and temporal reasoning. A motion-based method to identify video frames to be analyzed semantically, significantly reduces the number of frames forwarded to a conventional video frame analysis process and to enabling efficient processing on low-power hardware. A subset of the UCF-Crime dataset was used to evaluate the performance of proposed system. It consists of real-world surveillance video containing multiple examples of different types of anomalies. The results achieved a precision of 0.913, recall of 0.840 and F1 score of 0.875 while producing a 99.3% reduction in the number of frames sent to the vision-language model. Additionally, a human feedback loop allows the system to continuously adapt by changing how it asks for feedback and modifies future LoRA installations. Overall, the proposed framework demonstrates that the integration of multi-modality reasoning, efficient video understanding and continuous learning will support the design of the next generation of intelligent surveillance systems.