Skip to content

CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generation

Sep 2026 · 0 citations · 53 references
Computer Science

TL;DR

Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport (CMA-OT), a novel paradigm that leverages an external music expert to provide hierarchical supervision for the generator's latent features, bridging the semantic gap and enhancing representation learning is proposed.

Abstract

Dance-to-music (D2M) generation aims to synthesize music that is rhythmically and stylistically aligned with dance videos. A key challenge arises from the semantic mismatch between sparse dance cues, such as rhythm and style, and the dense information required for music composition, including structure, instrumentation, and expressive dynamics. Existing methods typically rely on these sparse cues and supervise only the final audio output, resulting in poorly learned music representations and generated music with limited musicality and structural coherence. To address these issues, we propose Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport (CMA-OT), a novel paradigm that leverages an external music expert to provide hierarchical supervision for the generator's latent features, bridging the semantic gap and enhancing representation learning. To effectively incorporate hierarchical supervision, we introduce a curriculum-guided multi-scale learning strategy that progressively transfers musical knowledge from the expert to the music generator, enabling stable and effective representation learning. Moreover, to accommodate the semantic and structural variations across different expert scales and achieve fine-grained alignment under temporal mismatch, we propose a scale-aware optimal transport alignment mechanism, which models soft correspondences between hierarchical expert representations and the generator's latent features. Extensive experiments on two datasets demonstrate that CMA-OT achieves state-of-the-art performance in rhythmic synchronization, perceptual quality, and overall music generation.

View source

Similar papers

Open access Aug 2026

ChoreDiffusion: Beat-Aware Diffusion for Music-to-Dance Generation

ChoreDiffusion is presented, a diffusion-based framework that integrates explicit beat guidance directly into the denoising process, and results indicate that embedding beat-aware guidance during generation, rather than applying it afterwards, is an effective route toward human-like musicality in music-driven choreogra...

Yufei Gao, Qiang Wu, Shuliang Zhu et al. · 0 citations
Sep 2026

Controllable Music-to-Dance Generation with Hierarchical Motion Representation Learning.

Current music-to-dance generation methods mainly rely on musical features, limiting precise control over generated movements. In particular, most existing methods with control mechanisms do not support example-based control, in which a user provides a reference motion sequence and the generated dances follow its fine-g...

Meng-Qi Liu, Hao Gao, Haolun Li et al. · 0 citations
Preprint Sep 2026

Where the Body Keeps the Beat: Structured Motion Conditioning and Music Dynamics Supervision for Dance-to-Music Generation

Dance-to-music (D2M) generation aims to synthesize musically plausible soundtracks whose temporal structure aligns with a given dance performance. Representative D2M methods do not explicitly distinguish motion cues across body parts and frequency bands, while standard flow matching lacks a dedicated objective for supe...

Chang-Chang Sun, Lu Cheng, Yan Yan · 0 citations
Preprint Sep 2026

MVWeaver: A Hierarchical Music Video Generation Agent with a Learned Song-to-Visual Bridge

Music videos are an important form of audiovisual expression in contemporary culture. They translate and extend the expressive content of songs through deliberate visual design. Existing automatic music video (MV) generation systems can generate visually plausible shots, yet often struggle with long-form coherence and...

Si-Fei Li, Min-Yan Luo, Xu Li et al. · 0 citations
Preprint Aug 2026

MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation

MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation, is introduced, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control.

Shuyu Li, Ke-Jun Zhang, Jia-He Lei et al. · 0 citations
Aug 2026

L-dance: LLM-guided motion intensity aware natural dance generation

Experimental results show that L-DANCE improves instruction alignment and controllability over single-pass and rule-based baselines whilst preserving motion quality and rhythmic consistency, showing the effectiveness of combining LLM-based planning with learned motion-intensity-aware selection for controllable music-dr...

Wonseop Shin, Yonghoon Jung, Mi-Young Lee et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.