Vorch-Human is presented, a unified human-centric generation framework built on a dual-stream audio-video diffusion transformer that augments the conventional noisy audio/noisy video interface with clean condition-audio and condition-video token groups and adapts to long-form audio-driven generation by training with clean latent prefixes and using the same frozen-prefix recurrence at inference.
Abstract
Human-centric audio-visual generation spans several closely related tasks: animating a person from driving speech, jointly generating speech and video from a voice reference, and synthesizing a scene from paired appearance and voice references. Existing systems commonly solve these tasks with separate models, even though they share the same target modalities and differ mainly in which observations are provided as conditions. We present Vorch-Human, a unified human-centric generation framework built on a dual-stream audio-video diffusion transformer. Vorch-Human augments the conventional noisy audio/noisy video interface with clean condition-audio and condition-video token groups. Per-token task embeddings, temporal position types, condition masks, and a shared multimodal prompt encoder allow driving speech, timbre examples, first frames, and subject images to be expressed within one model. To supply the supervision required by this interface, we develop a two-level data pipeline. Level 1 analyzes each clip with speech recognition, vocal separation, face detection and tracking, active-speaker and synchronization models, audio/visual speaker clustering, and multimodal caption correction; it produces subject-indexed speech, appearance, and timbre annotations. Level 2 links the same person across clips from a common source video and mines identity- and outfit-consistent reference images after face, body, quality, pose, and vision-language verification. Finally, we adapt Vorch-Human to long-form audio-driven generation by training with clean latent prefixes and using the same frozen-prefix recurrence at inference. Each segment contributes only its newly generated suffix, reducing boundary discontinuity and long-horizon identity drift. Experiments on short and five-minute generation demonstrate strong identity preservation, audio-visual synchronization, and temporal stability.
Vorch-Streamer, a post-training framework that addresses real-time long-form Text-to-Audio-Video (T2AV) streaming and enables real-time long-form avatar audio-video streaming, and bounded causal context and four-step denoising are presented.
Meng-Lin Han, Yang Ding, Yu-Lei Lu et al.· 0 citations
Vorch-Omni is presented, a unified multi-task framework for audio-visual synthesis based on an arbitrary-condition-to-arbitrary-output formulation that supports over 10 tasks, including text-to-video, text-to-audio-video, image- and reference-conditioned generation, temporal extension, audio-driven generation, video tr...
Vorch Team, Xiaoyu Chen, Yang Ding et al.· 0 citations
TemporalTalk is presented, which closes this gap with three training-time mechanisms: a face-masked temporal consistency loss with stop-gradient and delayed activation, a multi-head self-attention module that replaces the fixed convolutional attention over the audio context window, and cosine learning-rate annealing wi...
Nhan T. Huynh, Nguyen N. D. Tran, Tuấn Anh Huỳnh et al.· International Conference on...· 0 citations
By releasing the compact 7B generator and 2K Refiner, this work seeks to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.
Jia-Shu Zhu, Yan-Hao Zheng, Rui Tian et al.· 0 citations
Vorch-IR is presented, a unified framework that supports single- and dual-person identity replacement, with optional background replacement, in a single model, and an automatic data construction pipeline that synthesizes paired supervision for all four editing settings is developed.
Yao-Wei Wang, Xiaoyu Chen, Xin Ma et al.· 0 citations
HUG-VIS, a unified benchmark for Human-centered Understanding and Generation in Visual Intelligence, contains 8,400 seated half-body videos of 30 professional actors, each performing the same 280 emotion-action-prompt assignments under a controlled Mandarin studio protocol, with synchronized video, audio, text, and alp...
Fei Ma, Ze-Bang Cheng, Ming-Hui Li et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.