Design of a Wearable EMG–IMU Gesture Recognition System for Real-Time Sign Language Translation in Assistive Communication Applications

Abstract

Communication barriers have existed between the speech-and-hearing-impaired people and rest of the world due to the non-mainstream nature of sign language (SL). While there are proposed options to translate, most of them are costly and undeployable. The paper aims to create an affordable system to provide impaired users ability to communicate with normal people easily using text-to-speech (TTS) technology with the power of artificial intelligence (AI) and Internet of Things (IoT), powered by ESP32 to accurately predict SL. This is supported by dual channel electromyography (EMG) sensors and 6 axis gyrometer that is processed through filtering and feature extraction to give out high quality data for prediction. This system allows seamless communication enabled by MQTT and Wi-Fi connection from a wearable glove to collect required data that is used to train a Random Forest (RF) model. On a held-out validation set the system attained 92% accuracy; however, in on-field deployment this fell to about 53% (38–70% across the three gestures), so the system is presented as a feasibility prototype rather than a deployment-ready aid. As a starting point for this approach, future work will expand the dataset with more participants and gestures, complete a subject-independent evaluation, and refine the wearable device.