Skip to content
Preprint

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Aug 2026 · 0 citations · 68 references
Computer Science Engineering

TL;DR

Motion-Omni is presented, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech.

Abstract

An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

ECHO-G: Embodied Co-speech Humanoid mOtion Generation

Generating full-body co-speech motion for humanoid robots requires coordinating speech prosody, linguistic content, and embodiment-specific motion. To this end, we present ECHO-G, a framework jointly conditioned on speech audio and timed transcripts. Its Speech-Grounded Diffusion Transformer (SGDiT) combines frame-alig...

Yi-Zhao Li, Pu-Sen Gao, Ming Wang et al. · 0 citations
Preprint Aug 2026

SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning

SpeechGym is presented, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the...

Jia-Jun Fan, Jing-Yuan Li, Prashanth Gurunath Shivakumar et al. · 1 citation · ⚡1

DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech

D DuplexGen is presented, a dialogue synthesis framework that explicitly decouples content, timing, and acoustics and produces conversational dynamics closer to the real-dialogue reference distribution than the stitching baselines evaluated in this study.

Pengcheng Wang, Sheng Li, JiyiLi et al. · 0 citations
Preprint Sep 2026

CharDuplex: Building Character-Consistent Full-Duplex Spoken Dialogue Models

Full-duplex speech models are moving voice interaction beyond conventional turn-taking, yet natural conversation is shaped not only by when an agent speaks, but also by how it behaves as a conversational character. We present CharDuplex, a character-driven full-duplex speech model that combines real-time spoken interac...

Donghang Wu, Yi-Si Liu, Chen Chen et al. · 0 citations
#machine learning Preprint Oct 2026

Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first...

J. Ni, Ze-Kun Li, Zhiyang Dou et al. · 0 citations
Preprint Sep 2026

What Did I Just Say? Self-Listening for Full-Duplex Speech Models

Self-Listening is proposed, a full-duplex modeling approach that interleaves user speech, model text, and the model's played speech so that interruption recovery in what the user has actually heard is grounds in what the user has actually heard.

Xuan-Ning Zhou, Junyi Ao, Xiao-Tong Liu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.