GestNet: An Adaptive Approach to Multi-Modal Gesture Recognition With Semi-Supervised Learning Through EEG and sEMG Fusion

Abstract

The ability to reliably decode upper-limb motor intent from biosignals is a key requirement for next-generation Brain-Computer Interfaces (BCIs) and assistive neurorehabilitation systems. In this paper, the authors present GestNet, a semi-supervised deep learning framework for combining surface Electromyography (sEMG) and Electroencephalography (EEG) to achieve robust gesture recognition. This architecture uses a multi-branch Variational Autoencoder (VAE) backbone with modality-specific asymmetric encoders to learn disentangled latent representations of multimodal biosignals. The encoder-decoder pipeline includes convolutional layers, progressive downsampling, KL-divergence regularization, and a latent classifier for joint reconstruction and gesture prediction. A systematic temporal segmentation analysis over a range of window sizes (50–300 ms) reveals that 200 ms is the optimal window for the trade-off between temporal resolution and computational efficiency. The model shows superior performance, with extensive evaluation conducted on 11 healthy participants and seven upper-limb gestures, yielding 94.47%, 93.72%, 93.11%, 93.41%, and 96.45% on the test set for accuracy, precision, recall, F1-score, and AUROC, respectively. GestNet consistently outperforms state-of-the-art classifiers, such as CNNs, LSTMs, and hybrid CNN-LSTM models, in comparative studies. We test robustness using ablation studies with simulated Gaussian noise (10%) and signal dropout (30%). Multimodal fusion (EEG+sEMG) significantly outperforms unimodal (sEMG) methods, indicating that the modalities provide complementary information. GestNet provides a generalizable, interpretable, and noise-resistant method for gesture recognition from biosignals, with great promise for real-time neurorehabilitation and human-machine interaction, though it awaits further clinical population validation.