Hand Gesture Recognition Using MediaPipe Framework: A Vision-Based Touchless Interaction System

Abstract

Hand gesture recognition is a key enabling technology in vision-based touchless interaction systems, allowing users to interact with digital applications through natural hand movements without physical contact. Traditional input devices such as keyboards, mouse, and touchscreens present significant limitations in environments such as education, healthcare, and public infrastructure, where hygiene, accessibility, and ease of use are critical. MediaPipe, an open-source real-time framework developed by Google, provides accurate hand tracking by detecting and tracking 21 hand landmarks from live camera input using lightweight convolutional neural network models. This paper presents a comprehensive study of the working principles, modular pipeline architecture, feature extraction methodology, gesture recognition process, and experimental evaluation of MediaPipe-based hand tracking systems. A system-level architecture diagram is presented along with gesture classification techniques covering static and dynamic gesture modelling, SVM, k-NN, and LSTM-based approaches. Experimental results on a self-collected dataset of ten gesture classes across 30 participants demonstrate an overall recognition accuracy of 96.4%, a macro F1-score of 0.963, and a mean per-frame inference latency of 18.3 ms at 30 FPS. Comparative analysis confirms that the proposed system outperforms baseline CNN and SVM pipelines while operating on standard commodity hardware. The study demonstrates the potential of MediaPipe as a robust and accessible foundational technology for next-generation touchless interactive systems.