Deep Learning-Based Speech-to-Sign Language Translation for Inclusive Communication

Abstract

The developed system converts speech into sign language in real time, taking into account the mixture of Kazakh and Russian languages commonly used in Kazakhstan and other multilingual communities. Such bilingualism presents additional challenges for machine translation systems, making the development of adaptive models particularly relevant for inclusive communication. The purpose of this study is to develop an innovative real-time Speech-to-Sign Language Translation module for Kazakh-Russian Sign Language (KRSL) using an advanced neural network architecture. Speech is an essential means of communication, and its automatic translation into sign language enhances inclusiveness for people with hearing and speech impairments. The proposed system integrates Automatic Speech Recognition (ASR) and Text-to-Sign Language (T2S) conversion modules and employs Conformer and Transformer architectures for multilingual processing of Kazakh and Russian languages. In this study, a parallel corpus of speech, text, and sign language was collected and annotated, and a deep learning model was designed to convert speech utterances into sign language sequences while considering the spatial grammar and morphological characteristics of KRSL. The system is optimized for real-time performance, supports communication in multilingual environments such as Kazakhstan, and can be adapted for other bilingual regions and sign languages. Furthermore, the proposed approach contributes to the development of accessible and inclusive communication technologies.