CNN-ViT Hybrid Architecture With Time-Color Coding for Classifying Behavioral Events in Video Sequences

Abstract

Classifying behavioral events in video sequences requires modeling morphology and motion under severe data constraints. This paper presents a hybrid two-stream CNN-ViT architecture combining EfficientNetB4 for local spatial features with a Vision Transformer branch for global context. Time-color coding (TCC) transforms video frame sequences into a single RGB image encoding object trajectory, ensuring compatibility with standard image classifiers. A three-stage training strategy with successive unfreezing of EfficientNetB4 blocks enables stable transfer learning on small datasets. The approach is evaluated on two domains: laboratory rat behaviors (1 710 training images, 13 classes) and jet aircraft maneuvers (2 819 images, 8 classes), using a frame-level held-out split with 1 000-iteration bootstrapping. The full-capacity model (49.7M parameters) achieved 94.62% median accuracy on the biological domain. On the aeronautical domain, reducing model capacity by 62.5% improved accuracy from 18.8% to 70.0% on the full 8-class task by preventing memorization of stabilized-camera recording artifacts (TCC decoherence); removing one structurally anomalous class raised accuracy to 86.43% on the remaining 7 classes, with a 34.9% reduction in classification errors. All pairwise differences between the proposed architecture and ablated configurations were statistically significant, except for the RGB-only ablation in Domain 2 (p = 0.538-0.604 across accuracy and F1-score; bootstrap empirical test, B $=1~000$ ; p $\le 0.006$ for all remaining comparisons). These results confirm that the CNN-ViT architecture with TCC provides a practical framework for small-scale behavioral video analysis, offering guidelines for model capacity selection and dataset curation under resource constraints.