Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport (CMA-OT), a novel paradigm that leverages an external music expert to provide hierarchical supervision for the generator's latent features, bridging the semantic gap and enhancing representation learning is proposed.
Abstract
Dance-to-music (D2M) generation aims to synthesize music that is rhythmically and stylistically aligned with dance videos. A key challenge arises from the semantic mismatch between sparse dance cues, such as rhythm and style, and the dense information required for music composition, including structure, instrumentation, and expressive dynamics. Existing methods typically rely on these sparse cues and supervise only the final audio output, resulting in poorly learned music representations and generated music with limited musicality and structural coherence. To address these issues, we propose Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport (CMA-OT), a novel paradigm that leverages an external music expert to provide hierarchical supervision for the generator's latent features, bridging the semantic gap and enhancing representation learning. To effectively incorporate hierarchical supervision, we introduce a curriculum-guided multi-scale learning strategy that progressively transfers musical knowledge from the expert to the music generator, enabling stable and effective representation learning. Moreover, to accommodate the semantic and structural variations across different expert scales and achieve fine-grained alignment under temporal mismatch, we propose a scale-aware optimal transport alignment mechanism, which models soft correspondences between hierarchical expert representations and the generator's latent features. Extensive experiments on two datasets demonstrate that CMA-OT achieves state-of-the-art performance in rhythmic synchronization, perceptual quality, and overall music generation.
ChoreDiffusion is presented, a diffusion-based framework that integrates explicit beat guidance directly into the denoising process, and results indicate that embedding beat-aware guidance during generation, rather than applying it afterwards, is an effective route toward human-like musicality in music-driven choreogra...
Current music-to-dance generation methods mainly rely on musical features, limiting precise control over generated movements. In particular, most existing methods with control mechanisms do not support example-based control, in which a user provides a reference motion sequence and the generated dances follow its fine-g...
Meng-Qi Liu, Hao Gao, Haolun Li et al.· IEEE Transactions on Visuali...· 0 citations
Dance-to-music (D2M) generation aims to synthesize musically plausible soundtracks whose temporal structure aligns with a given dance performance. Representative D2M methods do not explicitly distinguish motion cues across body parts and frequency bands, while standard flow matching lacks a dedicated objective for supe...
Music videos are an important form of audiovisual expression in contemporary culture. They translate and extend the expressive content of songs through deliberate visual design. Existing automatic music video (MV) generation systems can generate visually plausible shots, yet often struggle with long-form coherence and...
MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation, is introduced, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control.
Shuyu Li, Ke-Jun Zhang, Jia-He Lei et al.· 0 citations
Experimental results show that L-DANCE improves instruction alignment and controllability over single-pass and rule-based baselines whilst preserving motion quality and rhythmic consistency, showing the effectiveness of combining LLM-based planning with learned motion-intensity-aware selection for controllable music-dr...
Wonseop Shin, Yonghoon Jung, Mi-Young Lee et al.· The Visual Computer· 0 citations