Dual-Mode Feature Fusion for Dynamic Gesture Recognition Using GRU-LSTM in Complex Backgrounds

Abstract

To address the problems of poor gesture segmentation, incomplete feature extraction, and low recognition accuracy caused by cluttered backgrounds, light intensity fluctuations, and skin-like background interference in dynamic gesture recognition, a dual-mode feature fusion dynamic gesture recognition method based on GRU-LSTM is proposed for complex backgrounds. First, adaptive threshold segmentation and motion gesture segmentation are combined to realize fine gesture segmentation, with the Intersection over Union (IoU) and Dice coefficient reaching 92.1% and 93.5%, respectively. For feature extraction, a dual-mode fusion strategy is adopted: the MediaPipe model extracts 21 hand joint local features (processed by relative coordinate transformation and mean-variance normalization), and an improved CNN embedded with Inception and CBAM attention modules extracts global image features. A GRU-LSTM combined network is designed for sequence learning, which reduces the computational complexity by 28% compared with a single LSTM and improves the fitting accuracy by 2.7% compared with a single GRU. The Softmax-SVM combined algorithm and a three-layer MLP are used for dual-mode feature combined classification. A dedicated Seven-Step Handwashing Technique (SWT) dataset is constructed in collaboration with Hubei Provincial People's Hospital, and the proposed method achieves an average recognition accuracy of 97.73% on this dataset (with Precision 97.21%, Recall 96.89%, and F1-score 97.05%). Comparative experiments show that the method outperforms state-of-the-art (SOTA) methods (e.g., 5.2% higher accuracy than LSTM and 9.93% higher than HOG+SVM), and the inference speed reaches 28 frames per second (fps) meeting real-time recognition requirements. Experimental results confirm that the method can accurately segment and recognize dynamic gestures in complex backgrounds, and has strong robustness to light and background changes.