Visual Gesture Recognition for Fluent Musical Human-Robot Turn-Taking

Abstract

Musical ensemble performance relies on implicit visual gestures, such as eye contact and head nodding, to negotiate fluid turn-taking. While the field of robotic musicianship has explored visual coordination, many existing systems treat gestures as explicit control interfaces or rely on audio-only cues that introduce significant latency. To address this, we present a multimodal turn-taking system for a robotic marimba player utilizing a markerless vision pipeline to process facial landmarks and hand keypoints in real time. We conducted a within-subjects study $(\mathbf{N} \boldsymbol{=} \mathbf{20})$ comparing a traditional audioonly silence-detection interaction against three visual-based interactions: eye contact, a sequential eye contact with a head nod, and a deliberate thumbs-up gesture. Objective evaluations confirmed our first hypothesis; all visual conditions significantly reduced turn transition latency compared to the audio baseline $(\mathbf{p}<\mathbf{0. 0 0 1})$. However, only the “natural” eye contact with a head nod condition required significantly higher mental effort (p=0.039) and resulted in the lowest overall participant satisfaction. Conversely, Eye contact only was rated as the most fluent and practical modality $(\mathbf{p}<\mathbf{0. 0 5})$, while the thumbs-up gesture and audio-only baseline provided the highest levels of predictability and satisfaction. These findings reveal a trade-off between social naturalness and system reliability, suggesting that for robotic musicians, simple, reliably detectable cues, whether natural eye contact or explicit gestures, are more essential for perceived fluency than the integration of complex, multi-modal social mimicry.