Skeleton-based action recognition using two-stream ST-GCN and lightweight ViT
Abstract
Aiming at the issues of insufficient spatio-temporal feature extraction, fixed topological structure of skeletal graphs, and inadequate modeling of long-range joint dependencies in skeleton-based action recognition, a method named 2s-ViTGCN is proposed, which integrates a two-stream spatial-temporal graph convolutional network (ST-GCN) with a lightweight Vision Transformer (ViT). The method constructs a skeleton dataset using a pose estimation algorithm and employs a dual-stream architecture that takes joint coordinates and bone vectors as input to fuse multi-modal information. By embedding a lightweight ViT module into the spatio-temporal graph convolution, dynamic adjacency matrix learning and long short-term spatio-temporal dependency modeling are achieved. Experiments on a self-constructed dataset demonstrate that the proposed method outperforms current mainstream models in recognition accuracy. The ablation tests indicate that the small ViT unit and two-stream structure are also beneficial to the performance. This paper can be used to increase the precision and reliability of skeleton based action recognition by building topology adaptively as well as using multi-modal features.