Cross-Modal Co-Attention With Event-Level Context Fusion for Abnormal Recognition

Abstract

The widespread integration of IoT sensing in intelligent campus environments creates a strong demand for behavior recognition systems that are both reliable in crowded, rapidly changing scenes and efficient enough for real-time deployment. However, conventional video-centric approaches often struggle to capture cross-modal cues and fine-grained event evolution, leading to degraded performance under realistic campus conditions. This study proposes cross-modal co-attention with event-level context fusion (CMCA-ECF), a multimodal framework that jointly models visual, audio, and textual streams collected by IoT infrastructures. CMCA-ECF projects heterogeneous modalities into a shared latent space and applies cross-modal co-attention to emphasize informative spatiotemporal correspondences. An event-level context fusion module further integrates temporal and scene-specific cues to better distinguish subtle precursors and follow-up actions associated with suspicious events. To enable practical deployment, a teacher–student distillation strategy transfers multimodal knowledge from a high-capacity teacher to a lightweight student for efficient inference. Extensive experiments on CABR50, with additional evaluations on UCF-101 and HMDB-51, demonstrate the accuracy and deployment efficiency. On CABR50, the distilled student CMCA-ECF-S achieves 87.34% Top-1 accuracy while running at 78 FPS. The teacher model CMCA-ECF-T reaches 91.57% Top-1 accuracy, providing a strong upper bound for distillation. Overall, the results indicate that CMCA-ECF offers a favorable balance between recognition performance and real-time throughput, with robust transferability across diverse benchmarks.