A Mobile-Based Real-Time Indonesian Sign Language (SIBI) Translation Application Using MediaPipe and LSTM

Abstract

Communication between deaf individuals and the hearing public remains difficult, since sign language is still unfamiliar to much of the general population. This paper presents an Android-based Indonesian Sign Language (SIBI) translation application built around a pre-trained Long Short-Term Memory (LSTM) model for gesture recognition. The application processes live camera input through MediaPipe, classifies gestures using TensorFlow Lite, and converts the output into text or speech. Development followed a Kanban workflow to support iterative implementation and the functional completion of individual features. In testing, the application was able to recognize SIBI alphabet gestures in real time and was responsive on mid-range Android devices. Functional testing also showed that core features were reliable. During a usability session with a deaf alumnus, we learned more about the practical strengths and limitations of the system, and in particular its difficulty in distinguishing gestures that look similar. Under these more stringent conditions of real use the estimated error rate was about 30%. The participant, however, responded positively to the application in terms of usability and potential usefulness. This is an early prototype of a mobile assistive communication aid. Subsequent steps will include increasing gesture coverage, improving recognition under difficult conditions, and testing with a larger group of users.