Real-Time Co-Speech Gesture Video Generation
Abstract
Co-speech gesture video generation aims to create realistic videos of speakers where facial expressions and body movements are naturally synchronized with the speech audio. An effective system must reconcile two often-conflicting objectives: a) real-time performance, which is crucial for interactive applications, and b) high-fidelity synthesis, ensuring photorealism and natural motion. While impressive progress has been made, most methods excel at one at the expense of the other. To bridge this gap, we present a two-stage framework that achieves zero-shot high-fidelity synthesis while operating in real-time. Our approach strategically employs lightweight 2D upper-body keypoints as an efficient intermediate representation, decoupling the task into two stages. First, a motion generation module predicts precise 2D keypoints from the audio input, explicitly capturing its rhythm and semantic cues. Second, a rendering module employs an implicit warping strategy to synthesize the final video. Conditioned on the predicted poses, it uses a cross-modal attention mechanism to warp a source speaker image to the target pose. Our proposed method supports real-time generation of 512x512 resolution videos at up to 30 frames per second on a single RTX 3090. To facilitate research, we also introduce TED-GVD (TED Gesture Video Dataset), a new large-scale dataset with 150 hours of high-quality upper-body speech videos and corresponding skeleton annotations. Extensive experiments demonstrate that our approach achieves visual realism and synchronization accuracy comparable or superior to state-of-the-art methods while offering substantial speedups over diffusion-based methods.