Nov 2026· Proceedings of the 6th International Workshop on Human-centric Multimedia Analysis· 0 citations· 14 references
TL;DR
A retrieval-based evaluation protocol, which leverages the intrinsic connection between retrieval and generation tasks and utilizes contrastive learning to encode cross-modal music-dance semantics, is introduced.
Abstract
Music-to-dance generation is a challenging and impactful task at the intersection of choreography, virtual reality, and generative content. However, traditional methods suffer from limited choreographic consistency. This oversight is reflected in unnatural limb movements, repetitive gestures, or overly stationary movements in the generated dances. To address these issues, we propose MATHDance, a novel architecture for music-driven 3D dance generation. By decoupling choreographic consistency into physical plausibility and aesthetic quality, MATHDance achieves significant performance improvements. It consists of two stages: (1) The High-Fidelity Dance Tokenization Stage (HFDT) pioneers the utilization of Finite Scalar Quantization (FSQ) for uniform tokenization. (2) The Hybrid Music-to-Dance Generation Stage (HMDG) employs a Mamba-Transformer hybrid architecture to autoregressively map music to dance tokens. Simultaneously, current evaluation metrics are based on handcrafted low-level features. Such limitations fail to capture the artistic essence of music and dance, thereby misleading the generative process toward producing dances that lack choreographic consistency. To address this, we introduce a retrieval-based evaluation protocol, which leverages the intrinsic connection between retrieval and generation tasks and utilizes contrastive learning to encode cross-modal music-dance semantics. Extensive experiments on FineDance and AIST++ demonstrate that MATHDance achieves state-of-the-art performance in both generation quality and computational efficiency, while validating the reliability of our evaluation protocol.
Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport (CMA-OT), a novel paradigm that leverages an external music expert to provide hierarchical supervision for the generator's latent features, bridging the semantic gap and enhancing representation learning is proposed.
Jin-Ting Wang, Chen-Xing Li, Dong Yu et al.· 0 citations
Multi-modal talking avatar synthesis aims to generate realistic talking videos from a reference portrait and speech. Despite rapid progress in diffusion-based methods, existing approaches still struggle to jointly achieve accurate lip articulation, human-preferred motion aesthetics, and efficient inference. We observe...
A multi-task learning framework based on diffusion models and spatio-temporal graph convolutional networks is proposed, which has the potential to approach the level of professional choreography in terms of artistic expressiveness and technical indicators.
This work introduces StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation, and achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among th...
Chengli Feng, Zhi-Yue Wu, Jia-Hao Song et al.· 1 citation
Driven by the advancement of artificial intelligence in music generation and cross-modal composition, achieving coherence of musical themes and accurate matching of background music in videos has become a research hotspot. To this end, this study constructs a music generation model with a multi-encoder Transformer and...
Ruo-Qi Wang· International Journal of Com...· 0 citations
With the in-depth application of artificial intelligence in music creation, the automatic generation of guitar chords based on deep learning still has shortcomings in melody-harmony consistency, long-term modeling, and playability. To address the aforementioned challenges, this study proposes a Melody-Constrained Bidir...
Qing-Ling Xu, Jing Wang, Zhao-Feng Xue et al.· International Journal of Aco...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.