FGA: Face Gesture Assistance
Abstract
Traditional desktop interfaces rely on a physical mouse and keyboard, which can be difficult to use for people with limited hand mobility and in hands-busy situations. Face Gesture Assistant (FGA) is an offline desktop interaction prototype that combines facial-landmark pointer movement with speech commands and dictation. The camera pipeline uses OpenCV and MediaPipe Face Landmarker to estimate a facial refer-ence point, map movement to screen coordinates, and smooth pointer motion. Mouth opening is detected using the Euclidean distance between upper- and lower-lip landmarks. The audio pipeline captures 16-kHz mono audio, uses preprocessing and Silero voice activity detection (VAD), and transcribes speech locally with Faster-Whisper. A command handler normalizes text and dispatches supported desktop operations. The camera and speech pipelines run in separate Qt threads within one application process. During detected speech, mouth-click actions are suppressed to reduce unintended clicks caused by speaking-related lip movement. Runtime logs from three Windows laptops show that effective camera rate and end-to-end latency vary by run, while cursor-coordinate processing is comparatively inex-pensive. These findings are engineering observations rather than a controlled accessibility or usability study. The paper describes the implementation, summarizes the available measurements, and identifies additional testing needed to establish reliability.