Vision-Driven framework For Interpreting Human Gestures into Natural Language

Abstract

Communication limitations between sign language and the hearing community have remained an impediment for inclusion and access of deaf and hard-of-hearing persons into vital and basic life requirements. Towards overcoming this major challenge, this paper advances a vision-based framework for interpreting and processing natural language from human gestures using computer vision and machine learning approaches. The proposed framework, chiefly, employs camera devices in capturing real-time gestures and understanding hand, body, and facial movements. Efficient techniques have been incorporated, such as detecting gestures and landmarks using OpenCV and MediaPipe, and further analyzing and processing gestures with landmark detection techniques and machine learning-based classification using TensorFlow. Experimental results indicate the accuracy and reliability of a proposed system for vision-based natural language translation. Thus, the proposed vision-based framework can be a potential and inclusive technology in addressing the challenges and limitations in natural language understanding and accessibility, towards promoting inclusive and equitable societies by ensuring accessibility for all, thus promoting and sustaining inclusive growth and social stability.