Design and Development of an AI-Powered Hand Gesture Recognition System for Touchless Screen Interaction

Abstract

The COVID-19 pandemic has accelerated the demand for touchless technologies that enhance public health and safety in shared environments. Hand gesture recognition offers an effective solution by enabling users to interact with digital devices without physical contact, thereby reducing the risk of contamination while improving the user experience. This paper presents the design and development of an AI-powered hand gesture recognition system for touchless screen interaction. The system was implemented in a Jupyter Notebook environment using OpenCV, MediaPipe, NumPy, and PyAutoGUI to perform real-time hand detection, gesture recognition, and virtual mouse control through a standard camera. The proposed system supports functions such as cursor movement, clicking, text selection, and scrolling using predefined hand gestures. The system was evaluated through usability testing involving IT professionals and students using a structured questionnaire covering accessibility, functionality, familiarization time, user interaction, physical contact avoidance, and overall satisfaction. The results showed that all respondents were able to download and use the system successfully, while the majority found the hand gesture dictionary helpful for learning the system. Scrolling and mouse cursor control were identified as the most useful features, and respondents unanimously agreed that the system effectively eliminated the need for physical contact. Overall, 61.9% of respondents rated the system 4 out of 5, indicating a high level of user satisfaction. The findings demonstrate that the proposed system provides an accessible, practical, and hygienic alternative to conventional input devices while promoting more natural human-computer interaction. Future enhancements will focus on expanding gesture functionalities and further improving system accuracy and usability.