EMG-CrossFormer is introduced, an end-to-end hybrid convolutional-transformer for seamless multimodal integration that improves sEMG-only decoding and that multimodal fusion substantially amplifies this benefit, underscoring the value of both design principles for complex hand gesture recognition.
Abstract
Hand gesture recognition via surface electromyography (sEMG) is fundamental to prosthetic control. In this field, deep learning approaches have become the gold standard. However, current architectures struggle to scale; model performance typically decreases as the number of hand movements increases. Performance degradation is tied to the increased statistical complexity of decoding expanded gesture sets and compounded by the limitations of state-of-the-art methods, which primarily rely on low-latency unimodal convolutional architectures. Convolutions operate locally, limiting model's ability to capture long-range sequential patterns. Unimodal setups cannot leverage complementary information from coordinated signals characterizing movement execution, such as inertial and eye-tracking data. These limitations motivate architectures that integrate local and global features across multimodal physiological sequences. To bridge this gap, this study introduces EMG-CrossFormer, an end-to-end hybrid convolutional-transformer for seamless multimodal integration. EMG-CrossFormer combines representations from an arbitrary number of unimodal encoders through cascaded cross-attention fusion layers, and decodes the fused representations using learnable gesture queries. EMG-CrossFormer was evaluated on four NinaPro datasets (DB2, DB3, DB7, and DB10) and benchmarked against six state-of-the-art models using an increasing number of modalities. Using only sEMG, EMG-CrossFormer achieved mean accuracies of 72.33%, 52.48%, 79.16%, and 73.49% on DB2, DB3, DB7, and DB10, respectively. Incorporating inertial signals improved performance to 90.66%, 80.40%, 92.79%, and 92.06%. These results show that joint local-global feature modeling improves sEMG-only decoding and that multimodal fusion substantially amplifies this benefit, underscoring the value of both design principles for complex hand gesture recognition.
Surface electromyography (sEMG) is a robust non-invasive modality for human-machine interaction, yet its application remains largely limited to coarse motor tasks such as grasping or rotation. The decoding of fine motor skills, specifically hand-writing, remains a challenging problem with potential relevance for prosthetic control and natural communication interfaces. In this work, we explore a Transformer-based alternative to classical signal-processing pipelines that treats multichannel sEMG signals as complex time series. We introduce DualMyo, a specialized model integrating Patch Embeddings and Rotary Positional Embeddings (RoPE) to capture the intricate spatiotemporal dynamics of myoelectric activity. Our experimental results show strong intra-session performance. Furthermore, we address the inherent challenges of signal drift and sensor displacement in cross-session applications. We show that a lightweight fine-tuning strategy of 10 epochs enables DualMyo to effectively adapt to session variability, achieving approximately 91% accuracy with two examples per digit. These findings provide a promising step toward adaptive sEMG-based hand-writing interfaces, although further validation is required for real-time and multi-subject deployment and neuromuscular control.
Maria Golitsyna, A. Makarova, M. Lebedev· bioRxiv· 0 citations
Decoding hand kinematics from surface electromyography (EMG) is a core challenge in wearable biosignal processing with clinical relevance for prosthetic control and motor rehabilitation. Most representation learning approaches for EMG focus on discrete gesture classification, and few focus on continuous regression. We present KinEMbed, a cross-modal contrastive learning framework for hand kinematics regression that jointly trains dual encoders -- one for windowed EMG features and one for kinematic (joint angle) targets. The resulting embeddings inherit the geometric structure of the kinematic space without requiring kinematic signals at inference time. Evaluating on the NinaPro DB8 dataset that includes both able-bodied users and subjects with limb difference (N=11), KinEMbed outperforms PCA, PLS, autoencoder and contrastive (CEBRA) baselines on held-out sessions, with largest gains on the most challenging thumb degrees of articulation. We position this work as a first step toward contrastive representation learning for regression of hand kinematics from structured wearable biosignals.
Sofia Gilardini, Chenfei Ma, K. Nazarpour· 0 citations
Predicting continuous finger kinematics from surface electromyography (sEMG) signals provides crucial input for the intuitive proportional control of extreme upper-limb robotic prostheses. However, this regression task remains a complex challenge due to the noisy and non-linear dynamics of muscle activations. In this study, a hybrid deep spatio-temporal model is proposed that combines a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) network to predict simultaneous finger joint angles from raw sEMG signals. Evaluated on 27 subjects from the Ninapro DB1 dataset using a rigorous repetition-wise split, the model was tested across three input window sizes (400ms, 600ms, and 800ms). Results demonstrate that the Hybrid CNN-LSTM consistently outperforms standard LSTM and Bidirectional LSTM (BiLSTM) baselines. The optimal 800ms window achieved the best overall performance (R2 = 0.7733, PCC =0.8841, RMSE =11.65°), demonstrating high-fidelity tracking particularly in the highly active digits. These findings highlight the feasibility of deploying efficient, single-modality deep learning estimators for robust, real-time prosthetic control without requiring complex multimodal sensor fusion.
Sukrit Ghosh, Yashkrit Singh, K. K. Sah et al.· International Conference on...· 0 citations
Accurate classification of hand gestures from surface electromyography (sEMG) signals is essential for human-computer interaction and myoelectric prosthetic control, yet conventional feature-extraction methods struggle to capture the temporal dynamics of muscle activation. Dynamic Mode Decomposition (DMD), a data-driven technique originally developed for fluid dynamics, has not been extensively validated for sEMG signals. This study addresses that gap by introducing DMD as a feature-extraction framework for sEMG-based gesture classification, providing a mathematically rigorous approach for capturing spatiotemporal muscle-activation patterns. Using sEMG recordings from 37 participants, we extracted DMD-reconstructed features–Mean Absolute Value (MAV), Root Mean Square (RMS), Simple Square Integral (SSI), Variance (VAR), Standard Deviation (STD), Median, Integrated EMG (IEMG), and spectral features–and compared their discriminative power against traditional features across multiple classifiers. DMD-derived features significantly distinguished common hand gestures ( $p \lt 0.05$ , Bonferroni-corrected) and consistently outperformed traditional features: MAV, RMS, SSI, and VAR achieved 96% accuracy with k-nearest neighbors and 88% with random forest, while support vector machines reached 97% and 96% accuracy using STD, MAV, and RMS, respectively. Convolutional Neural Networks and Deep Stacked Neural Networks achieved up to 85% and 87% accuracy, respectively, using DMD-extracted spectral features. These results establish DMD as an effective, interpretable feature-extraction method for sEMG-based gesture classification, with applications in rehabilitation engineering, prosthetic control, and human-computer interaction.
Alberta Ashitey, Williams Ayivi, Joan Amos Toluwani et al.· IEEE Access· 0 citations
This study investigates the feasibility of classifying ten hand gestures using a single sEMG channel combined with lightweight machine learning architectures and demonstrates that combining time and frequency features with Pearson filtering and a compact NN can achieve up to 90 percent accuracy.
A three-branch fusion network that explicitly models the ring arrangement of armband electrodes, capturing the adjacency information in the sensor topology that linear channel representations ignore and generalizes to MyoArmbandDataset under a subject-adaptive transfer learning protocol without dataset-specific hyperparameter retuning.
Luoqi Cui, Yong Liu, Hadi Fathollahi Abdar et al.· Italian National Conference...· 0 citations