Skip to content
Preprint

CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation

Oct 2026 · 0 citations · 21 references
Computer Science Engineering

TL;DR

CAS (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents, which is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.

Abstract

General-purpose self-supervised speech encoders provide strong acoustic representations, but their pretraining objectives are not explicitly aligned with the visible articulatory dynamics required by speech-driven 3D facial animation. We propose CASE (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents. After adaptation, the geometry branch is removed, and the audio-only CASE representation is fused with the baseline SSL speech representation through residual cross-attention, while retaining the motion-decoder architecture and objective. Across four VOCASET baselines, CASE reduces mean lip vertex error by 2.8-6.0%, and mean full-face vertex error decreases in every system. Under the same fusion protocol, CASE yields lower mean LVE than the unadapted AV-SE stream on all four generators, with model-dependent margins. Because adaptation uses Korean speech-geometry pairs and evaluation uses English VOCASET, the result is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.

View source

Similar papers

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

A novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion and integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion.

H. Kim, Byungchan Hwang, H. Kim · 0 citations
Oct 2026

Audio-driven 3D Facial Animation with Multimodal Context Enhancement

Audio-driven 3D facial animation has emerged as a prominent research topic in computer graphics. Existing autoregressive methods are often slow and suffer from error accumulation, whereas non-autoregressive methods often struggle to establish global contextual dependencies. Moreover, current approaches remain insuffici...

Yan-Lei Lei, Jing Chi, Chun-Lin Zhao et al. · 0 citations
Preprint Oct 2026

Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioning

Video-to-speech synthesis aims to generate natural-sounding speech from silent talking-face videos while ensuring phonetic accuracy. A fundamental challenge in this task is the inherent one-to-many mapping problem, where visual dynamics often lack sufficient information to uniquely determine the corresponding utterance...

Gun-Woo Lee, Yoori Oh, Yoseob Han · 0 citations
Preprint Oct 2026

Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visual speech representation learning via cross-modal distillation and joint face-lip modeling. FLARE leverages semantically...

Jing-Xuan Zhang, Gen-Shun Wan, Jian-Qing Gao et al. · 0 citations
Open access Oct 2026

PAGE: Parts-Aware GuidancE for Co-Speech Gesture Portrait Video Generation

Co-speech gesture video generation aims to synthesize realistic, audio-driven portrait videos from a single reference image and a speech signal. Despite advances in visual quality, diffusion-based approaches still suffer from degraded local details and imperfect audio-visual alignment. We present PAGE (Parts-Aware Guid...

Ya-Biao Wang, Jia-Fu Wu, Y. Xu et al. · 0 citations
Nov 2026

PersonaDiff: diffusion-based style control for speech-synchronized 3D facial animation

Qualitative results show that the proposed dynamic style representation enables the generation of diverse speaker-dependent facial motions under different speaking styles and is well suited for applications such as virtual avatars, digital content creation, and other digital human systems.

Wei Li, Yu-Pu Song, Ya-Zhi Liu · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.