An Attention-Guided Hybrid CNN–Vision Transformer Framework for Real-Time and Edge-Deployable Crowd Density Estimation

Abstract

Estimating crowd densities accurately is still a key issue in the field of intelligent surveillance systems, especially in developing more advanced Deep Learning models which can be implemented on edge devices with limited resources. Recent Vision Transformer (ViT) models have excelled in capturing long-range global relationships that are hard for other models, but at a high computational cost, and are often prohibitive for real time use in practical settings. In this paper, we propose a novel attention-guided hybrid framework which integrates both the ability of Convolutional Neural Networks (CNNs) to capture local features and the ability of Vision Transformer (ViT) for modeling contexts globally to obtain the best of both worlds. With that in mind, our architecture is a dual-branch one, consisting of a Lightweight CNN branch to capture fine-grained spatial details and local patterns, with a Compact ViT branch to model long-range dependencies and global scene understanding. They are intelligently fused via an Attention-Guided Feature Fusion Module (AGFFM) that dynamically weights multi-scale features with both spatial and channel attentions. For easy edge deployment, we extensively optimize the pipeline, which utilizes INT8 quantization, structured pruning, and knowledge distillation to reduce the model size by 65% and computational expenses by 58%. Our framework achieves competitive accuracy (e.g., MAE: 68.2 on ShanghaiTech Part A) and remains real-time at 28.5 FPS on NVIDIA Jetson Nano in extensive experiments on three benchmarks: ShanghaiTech Part A/B, UCF-QNRF and NWPU-Crowd. The proposed framework is a stepping stone from the state of the art in terms of accuracy so far to deployable for real-life crowd monitoring application.