This work presents EMODY Flow, a lightweight flow-matching framework that attaches to a frozen Qwen-3 Omni model and reuses its internal Mimi audio-codecs to condition two parallel DiT generators - one for SMPL-X body pose, one for FLAME facial expressions.
Abstract
Embodied conversational agents require synchronized full-body motion (body gestures and facial expressions) that aligns with speech and emotional state. Omni-modal large language models excel at multimodal understanding but produce only linguistic outputs, leaving a critical gap in embodied response generation. We identify and address a failure of emotion conditioning: like other conditional generators that under-use weak conditioning signals, a flow-matching model given both a rich audio embedding and a discrete emotion label suppresses the emotion, generating near-identical motion regardless of the specified emotion. We present EMODY Flow, a lightweight (around 35M parameters) flow-matching framework that attaches to a frozen Qwen-3 Omni model and reuses its internal Mimi audio-codecs to condition two parallel DiT generators - one for SMPL-X body pose, one for FLAME facial expressions. A training-time auxiliary emotion classifier restores emotion sensitivity by forcing generated motion to be emotion-identifiable. EMODY Flow sets a new state of the art on BEAT2 gesture quality, with FGD 0.302, Beat Correlation 0.853, and Diversity 24.62 - improving over the best prior results by 26%, 5%, and 62% respectively - and transfers to zero-shot facial animation on TFHP without domain-specific fine-tuning. Beyond these quantitative gains, the classifier yields clearly emotion-separated motion, which we demonstrate qualitatively through a multidimensional-scaling analysis of the generated gestures.
Generating full-body co-speech motion for humanoid robots requires coordinating speech prosody, linguistic content, and embodiment-specific motion. To this end, we present ECHO-G, a framework jointly conditioned on speech audio and timed transcripts. Its Speech-Grounded Diffusion Transformer (SGDiT) combines frame-alig...
Yi-Zhao Li, Pu-Sen Gao, Ming Wang et al.· 0 citations
A multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions, and dual-stage Valence-Arousal-Dominance conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings.
Tian-Chi Liu, Ze-Yang Song, Tian-Rui Wang et al.· 6 citations
SteerSpeech is presented, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen.
A. Benazir, Darius Petermann, Felix Xiaozhu Lin et al.· 0 citations
Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computation and emotion-labeled speech training data. We therefore study vector steering, a training-free approach that modifies the internal represe...
Kuan-Po Huang, Hao-He Liu, Pu-Yuan Peng et al.· 0 citations
Detecting true felt emotions when speakers suppress or mask their internal state poses a fundamental challenge for affective computing systems. We study a specific form of affective dissonance in dyadic speech: utterances where a speaker’s self-reported emotion diverges from all external observer ratings, indicating th...
Cheng-Shiuan Lin, T. Eze, Da-Wei Xie et al.· Proceedings of the 28th Inte...· 0 citations
Perceptual evaluations further show that training with AffectMoCap improves affect recognition from generated body motion, while real-robot experiments demonstrate continuous affect-conditioned behavior and stable long-horizon execution.
Cheng-Qun Yang, Teng-Jie Zhu, Liang Xu et al.· 0 citations