Motion-Causal Token Selection for Cross-Dataset Action Recognition

Abstract

Action recognition in video data has achieved remarkable success under the traditional intra-dataset evaluation paradigm, where training and testing data are drawn from identical distributions. However, deploying these models in real-world scenarios frequently results in severe performance degradation due to domain shifts across datasets. Current vision transformer architectures attempt to mitigate computational overhead and improve generalization through token selection mechanisms, but these methods often inadvertently preserve background tokens that exhibit strong, spurious correlations with specific action classes, thereby exacerbating the domain shift problem. To address this fundamental limitation, this paper introduces a novel framework termed Motion-Causal Token Selection for cross-dataset action recognition. By formulating the video representation learning process through the lens of structural causal models, we identify the dataset-specific background context as a primary confounder that creates spurious links between visual features and action labels. We propose a motion-guided intervention mechanism that leverages backdoor adjustment to decouple true motion-centric causal features from domain-specific contextual biases. Furthermore, a dynamic token scoring network is integrated to selectively retain tokens that exhibit high causal influence on the action category while discarding those driven by confounders. Extensive experiments on standard cross-dataset benchmarks, including Kinetics to UCF101 and HMDB51, demonstrate that the proposed method significantly outperforms existing state-of-the-art domain generalization approaches. The framework not only enhances cross-dataset transferability but also reduces the computational footprint of vision transformers during inference.