In computational neuroscience, the statistical nature of primate visual responses has long served as a benchmark for efficient coding. Specifically, previous works demonstrated that in the primate Anterior Inferotemporal (AIT) cortex, population sparseness $(S_{p})$ significantly exceeds single-neuron sparseness $\left(S_{l}\right)$, demonstrating that while individual neurons respond to relatively simple features, the total available feature space is vast. In this work, we establish a comparative experimental framework to bridge the gap between biological neural responses and the internal representations of Vision Transformers (ViTs) and ResNet-50. We analyze layer-wise kurtosis dynamics across ViT-B/16, ViT-L/16, ViT-B/32, and ResNet-50 using 3000 images of ImageNet-1k validation and Caltech-101 datasets. Our results reveal a “Semantic Snap” that is not a fluke and that it is a robust architectural phenomenon of visual models. Notably, while ResNet-50 inverts the biological signature $(S_{l}>S_{p})$, high-capacity transformer-based models like ViT-L/16 achieve a Lehky ratio, mirroring the distributed manifold coding of the AIT cortex. We further identify a significant magnitude gap in representational bandwidth between attention-based and convolutional architectures. These findings, validated by Pareto tail analysis, robust t-statistics, and Lehky ratio calculations, provide a computational link between transformer scaling laws and the Population Manifold Hypothesis in biological vision.
Ankit Sharma, S. Channappayya· International Conference on...· 0 citations
Action prediction from frames and videos is a well-studied problem. Models trained with a single modality, mostly vision, will fail in low-light conditions. Recent works have attempted to predict action categories using vision-language and audio-visual models. A challenge, however, is that some dataset annotations lack temporal ground truth and include only the vision modality. Relying on transformers and an intelligent Vision Language Model (VLM) is a viable solution, but deploying them on edge devices could lead to reduced performance and hallucinations. This work presents an Audio-Visual-Text (AVT)-based multi-step Pipeline for Action Prediction and Captioning (AVT-PAC) to address this problem. First, for an input video, we identify the area to focus on using the Region-of-Interest (ROI) Extraction module. CLIP and CLAP encoders are used for ROI prediction. However, the ROI extracted region may vary in duration, resulting in a large number of frames to be processed. To avoid learning from redundant frames, we uniformly sample key frames within the ROI extracted region using a keyframe extraction module. These keyframes are then used to train an Audio-Visual Action and Text-Aware Representation (AVATAR) model to predict actions and captions. Through systematic experiments, we demonstrated that the proposed AVATAR-TCN model beats the present state-of-the-art (SOTA) baselines on the AVE dataset. Code is available in https://github.com/Ifovia/AVT-PAC
A. R, Ambarish Parthasarathy, Sucharitha Devarakonda et al.· International Conference on...· 0 citations
Understanding long-range videos remains a key challenge in computer vision due to high temporal redundancy and computational burden. Despite strong performance of recent models, they are constrained in terms of scalability and generalization when applied to longer video sequences. In this work, we present Keyframe-based Spatio-Temporal Adaptive Representation (K-STAR), a redundancy-aware video summarization framework designed to generate compact and semantically rich representations that are effective in downstream tasks. The proposed method jointly models appearance and motion cues while filtering redundant frames. Importantly, it preserves critical temporal transitions while significantly reducing the number of processed frames. Additionally, each key frame is encoded using object, scene, and background-aware prompts, enabling richer semantic representation. Evaluated on the UCF-101 dataset, K-STAR achieves Top-1 accuracy of 93.06% and Top-5 accuracy of $\mathbf{9 8. 7 3 \%}$, with $\mathbf{5 6} \times$ frame reduction and $\mathbf{1 1. 5} \times$ faster inference, demonstrating competitive performance with substantially improved efficiency.
Rahul Kumar, S. Channappayya· International Conference on...· 0 citations
