Aug 2026· IEEE Transactions on Visualization and Computer Graphics· Vol PP, pp. 1-14· 2 citations· 69 references
Computer ScienceMedicine
TL;DR
A comprehensive framework that learns an automatic perceptual model from human preference data and leverages it to improve and evaluate the perceptual quality of audio-driven 3D facial animation and introduces Facial Motion reward Feedback Learning (FMFL), a direct fine-tuning algorithm that leverages a pretrained reward model to optimize diffusion-based audio-driven 3D facial animation models for better alignment with human preferences.
Abstract
Audio-driven 3D facial animation is essential for advancing immersion and interactivity in virtual experiences. Although recent advances have shown promising capabilities, the training and evaluation of existing methods typically rely on ground-truth-based errors, which fall short of aligning with human preferences. To address this, we present a comprehensive framework that learns an automatic perceptual model from human preference data and leverages it to improve and evaluate the perceptual quality of audio-driven 3D facial animation. To begin with, we construct FMPair (Facial Motion Pairwise preference), the first human preference dataset for audio-driven 3D facial animation, which is built through a systematic annotation pipeline and comprises 65,574 annotated 3D facial motion pairs from 8,834 distinct in-the-wild audio clips. Based on the pairwise comparison dataset, we propose a Facial Motion Reward model, termed FMReward, which takes audio and 3D facial motion as inputs and predicts a perceptual quality score aligned with human preferences. Building upon FMReward, we further introduce Facial Motion reward Feedback Learning (FMFL), a direct fine-tuning algorithm that leverages a pretrained reward model to optimize diffusion-based audio-driven 3D facial animation models for better alignment with human preferences. Extensive experiments demonstrate the superiority of FMReward over other metrics in aligning with human preferences and the effectiveness of FMFL in improving the perceptual quality of audio-driven 3D facial animation. The dataset and codes will be released at: https://github.com/wsj-sjtu/FMReward.
Audio-driven 3D facial animation has emerged as a prominent research topic in computer graphics. Existing autoregressive methods are often slow and suffer from error accumulation, whereas non-autoregressive methods often struggle to establish global contextual dependencies. Moreover, current approaches remain insuffici...
Yan-Lei Lei, Jing Chi, Chun-Lin Zhao et al.· ACM Transactions on Multimed...· 0 citations
Qualitative results show that the proposed dynamic style representation enables the generation of diverse speaker-dependent facial motions under different speaking styles and is well suited for applications such as virtual avatars, digital content creation, and other digital human systems.
Experimental results show that the complete landmark representation provides the best overall image-generation performance among the evaluated configurations, while Post-Refinement consistently improves frame-level visual quality.
Zi-Kai Shu, Takashi Nose, A. Ito· IEEE Access· 0 citations
A novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion and integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion.
H. Kim, Byungchan Hwang, H. Kim· arXiv.org· 0 citations
CAS (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents, which is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.
Jaehong Myung, Seungho Eum, Taegon Kim et al.· 0 citations
Although recent advances in AI-based Text-to-Motion (T2M) algorithms have enabled text-driven 3D Animated Human (3DAH), often referred to as "Puppeteers" for their ability to streamline 3D animation design, the quality of the generated motion remains a critical concern. More importantly, the T2M field continues to lack...
Ying-Jie Zhou, Rong-Hui Li, Zhao-He Lv et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.