Frame Importance Voting for Video Scene Classification

Abstract

Classification of video data is challenging due to the temporally repeated measures and the differential importance of different frames in a video clip. This paper proposes a framework called Frame Importance Voting (FIV), where frame importance weighting and voting are merged as part of a shared inference process to enhance the temporal classification of video data without additional computational burden. Spatial video features were extracted from the video clips using a ResNet-50 architecture, and the temporal relationships were modeled using a two-layer transform encoder. Frame significance was derived adaptively by summing the transformer's attention and confidence scores for each frame, and predictions for categories were made by summing the frame predictions using weighted voting. Results on Kinetics-400 (50 categories, 10,000 clips, 16-32 frames per video) confirmed that FIV achieved 77.4% top-1, 92.5% top-5, and 76.8% top-1, outperforming the aggregators by up to 5.6% (using about 37 million parameters).