Empirical Limitations of Gesture-Driven Air-Writing for Mathematical Symbol Recognition on Edge Devices

Abstract

Advances in computer vision have made contactless, hardware-free interaction with computers increasingly viable, with applications in education, medicine, and virtual reality. However, high classification accuracy on curated benchmark datasets may not reliably reflect real-world usability. This study evaluates that gap for gesture-based mathematical symbol recognition. We developed a gesture-based digital whiteboard that captures air-written strokes using MediaPipe hand tracking, applies aspect-ratio-preserving preprocessing, and classifies 54 mathematical symbol classes using a Convolutional Neural Network (CNN), benchmarked against k-NN and SVM baselines with PCA dimensionality reduction and Optuna-tuned hyperparameters. Targeted scale, shear, and morphological stroke-width augmentation were applied to improve robustness to real-world drawing variability. The tuned CNN achieved 85.2% accuracy on the full held-out test set, with an average classifier-only CPU inference time of 2.59 ms and a measured end-to-end frame latency of 33.8 ms on commodity edge hardware. A structured six-participant usability study comprising 1,944 isolated-symbol trials and 264 sequence-writing events revealed a substantial offline-to-live performance gap: overall live recognition accuracy was 5.9% (95% CI 4.9–7.0%). Accuracy was statistically indistinguishable between air-writing and mouse-drawn input (5.7% vs. 6.1%), while 48.7% of live predictions collapsed onto a single class at low mean confidence. These findings indicate that domain shift between offline training data and live-captured strokes is the dominant limitation, rather than air-writing-specific fatigue or tracking noise. Further domain adaptation or additional input modalities are therefore required before reliable practical deployment.