The Recognition-Consequence Gap: How Gesture Recognition Misses Human Action

Abstract

As computational systems that process multimodal behavioral streams (e.g., speech, gaze, facial expressions, gestures, and bodily movements) become increasingly capable, the field faces a critical question: does more embodied data necessarily yield deeper human understanding? This question is most visible in the gesture recognition community, where machine learning pipelines classify individual gestures against fixed taxonomies: such systems may recognize bodily signals while missing the situated actions those signals help accomplish. This is the recognition-consequence gap, the ontological divide between classifying bodily movements moment-to-moment and understanding what embodied action accomplishes in interaction. Drawing on social science perspectives that treat gesture as a cognitive, communicative, and interactional resource, gesture consequence is proposed as a theoretical reorientation for machine interpretation of gestures. Gesture consequence shifts attention from the classification of individual movements to the unfolding effects of embodied action across interaction, asking from a broader lens how gestures are taken up, ignored, repaired, mirrored, or transformed over time in social interaction. When multimodal systems classify gestures without account for how they are taken up in human interaction, these systems risk producing narrow accounts of human activity that leave out the meanings and forms of participation that unfold through the body.