SIMPLIFYING VIDEO-BASED SIGN LANGUAGE RECOGNITION: A COMPOSITE IMAGE METHOD FOR VISION TRANSFORMER DEIT ARCHITECTURE

Abstract

Context. The fundamental need for Sign Language Recognition (SLR) arises from the persistent communication gap between the deaf/hard-of-hearing community and the non-signing public. Traditional SLR models, however, are often hindered by the high computational complexity and overhead associated with modeling the temporal dynamics of video data (e.g., LSTMs or Video Transformers).Objective. The goal of this study is to develop a computationally efficient yet highly accurate SLR framework that effectively captures temporal information without relying on resource-intensive sequence models.Method. The proposed approach introduces a novel preprocessing pipeline that converts temporal video data into compact spatial representations. Specifically, ten equally spaced frames are extracted from each sign video and concatenated horizontally into a single composite image, substantially reducing data size and model complexity. The transformed image dataset is then utilized to train the facebook/deit-base-patch16-224 Vision Transformer (DeiT), optimized for efficient visual feature extraction.Results. The proposed DeiT-based framework achieved an overall classification accuracy of 98%, with a Hamming Loss of 0.0000 and near-perfect Area Under the Curve (AUC) scores across all eight classes. These results demonstrate its superiorperformance and computational efficiency compared to existing SLR models.Conclusions. The findings confirm that converting temporal video sequences into composite spatial representations enables a powerful yet lightweight SLR system. The proposed model offers a practical and deployable solution for isolated sign word recognition, achieving state-of-the-art performance with minimal computational cost. This framework paves the way for real-time, accessible, and scalable applications in assistive communication technologies