Continuous wrist kinematics carry lexical cues for temporal deixis

Abstract

Human communication unfolds as a continuous multimodal stream in which speech, prosody, and gesture are tightly coordinated. Yet gesture is often analyzed as a sequence of discrete, manually segmented events, separated from lexical semantics. This study asks whether subtle continuous wrist movements co-ocurring with spoken temporal adverbs carry enough information to support above-chance discrimination among lexical categories without manual gesture annotation. Using a large aligned corpus of English television news, we extracted 13,061 clips containing the words yesterday, today, and tomorrow. After active-speaker verification, shot-change filtering, speaker masking, and pose-visibility checks, we represented each target token as an 80-frame wrist-trajectory sequence centered on the word midpoint. A bidirectional LSTM trained only on wrist kinematic features classified the three adverbs with 0.49 accuracy, above the 0.33 chance baseline and above a shuffled-label control. Binary tests showed a strong asymmetry: tomorrow was more discriminable than yesterday or today, reaching 0.697 accuracy in a one-versus-rest task. Feature ablations showed that performance dropped when the model was restricted to raw coordinates or to one hand, suggesting that the observed lexical cues are distributed across richer kinematic descriptors rather than raw wrist position alone. These preliminary results support a continuous, multi-flow account of multimodal interaction in which low-level bodily motion provides weak but measurable probabilistic cues associated with lexical meaning.