SIGMA IoT enabled sensor based embedded gesture recognition system using RISP algorithm for real time assistive communication for speech and hearing disabilities

Abstract

Communication barriers faced by individuals with speech and hearing impairments continue to limit effective interaction and access to essential services. This paper presents SIGMA, an IoT-based assistive communication system implemented using a wearable smart glove equipped with flex sensors for real-time gesture acquisition. The proposed system enables bi-directional communication by translating hand gestures into on-screen text and synthesized speech, while also supporting speech-to-gesture visualization through a mobile interface. The proposed RISP algorithm is a lightweight gesture recognition approach that combines signal smoothing, thresholding, binary vector formation, and Hamming-distance matching for efficient real-time gesture classification on the ESP32 platform. The system is implemented on an ESP32 platform and communicates wirelessly via Bluetooth for low-latency interaction. An integrated GSM–GPS SOS module is included to enable emergency alert transmission with live location sharing. Experimental evaluation using a real-time primary gesture dataset demonstrates a gesture recognition accuracy of 96.2%, with an average response time of approximately 230 ms and an end-to-end latency of about 250 ms. The system provides synchronized multi-modal outputs, including text, synthesized speech, and visual gesture representation, demonstrating reliable performance under the controlled experimental conditions. The proposed glove is low-cost and portable, showing potential for assistive communication in education, healthcare, and daily communication environments. Further validation with larger and more diverse user groups is required before broader real-world deployment.