CAS (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents, which is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.
Abstract
General-purpose self-supervised speech encoders provide strong acoustic representations, but their pretraining objectives are not explicitly aligned with the visible articulatory dynamics required by speech-driven 3D facial animation. We propose CASE (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents. After adaptation, the geometry branch is removed, and the audio-only CASE representation is fused with the baseline SSL speech representation through residual cross-attention, while retaining the motion-decoder architecture and objective. Across four VOCASET baselines, CASE reduces mean lip vertex error by 2.8-6.0%, and mean full-face vertex error decreases in every system. Under the same fusion protocol, CASE yields lower mean LVE than the unadapted AV-SE stream on all four generators, with model-dependent margins. Because adaptation uses Korean speech-geometry pairs and evaluation uses English VOCASET, the result is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.
A novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion and integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion.
H. Kim, Byungchan Hwang, H. Kim· arXiv.org· 0 citations
Audio-driven 3D facial animation has emerged as a prominent research topic in computer graphics. Existing autoregressive methods are often slow and suffer from error accumulation, whereas non-autoregressive methods often struggle to establish global contextual dependencies. Moreover, current approaches remain insuffici...
Yan-Lei Lei, Jing Chi, Chun-Lin Zhao et al.· ACM Transactions on Multimed...· 0 citations
Video-to-speech synthesis aims to generate natural-sounding speech from silent talking-face videos while ensuring phonetic accuracy. A fundamental challenge in this task is the inherent one-to-many mapping problem, where visual dynamics often lack sufficient information to uniquely determine the corresponding utterance...
Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visual speech representation learning via cross-modal distillation and joint face-lip modeling. FLARE leverages semantically...
Jing-Xuan Zhang, Gen-Shun Wan, Jian-Qing Gao et al.· 0 citations
Co-speech gesture video generation aims to synthesize realistic, audio-driven portrait videos from a single reference image and a speech signal. Despite advances in visual quality, diffusion-based approaches still suffer from degraded local details and imperfect audio-visual alignment. We present PAGE (Parts-Aware Guid...
Ya-Biao Wang, Jia-Fu Wu, Y. Xu et al.· IEEE transactions on circuit...· 0 citations
Qualitative results show that the proposed dynamic style representation enables the generation of diverse speaker-dependent facial motions under different speaking styles and is well suited for applications such as virtual avatars, digital content creation, and other digital human systems.