Aug 2026· IEEE Transactions on Pattern Analysis and Machine Intelligence· Vol PP· 0 citations
Medicine
TL;DR
This work investigates a simple yet effective conditional generative framework for text-to-motion generation and proposes T2M-GIT+, which employs a non-autoregressive method to generate discrete motion representations in parallel, and is therefore more efficient than T2M-GPT+ while achieving comparable results.
Abstract
Based on Vector Quantized Variational AutoEncoder (VQ-VAE) and Transformers, we investigate a simple yet effective conditional generative framework for text-to-motion generation. First, we find that, with a few training recipes (EMA and Code Reset), a CNN-based VQ-VAE can learn high-quality discrete motion representations, which largely facilitate motion generation. Second, to better understand motion descriptions, we propose T2M-GPT+ that uses Large Language Models (LLMs) to extract text features. Further, we propose T2M-GIT+, which employs a non-autoregressive method to generate discrete motion representations in parallel, and is therefore more efficient than T2M-GPT+ while achieving comparable results. Comprehensive experiments show that our method is superior to existing methods, including competitive diffusion methods. For example, we achieve a Top-1 R-precision of 0.53 and an FID of 0.10 on the HumanML3D dataset, substantially outperforming MotionDiffuse, which obtains a Top-1 R-Precision of 0.49 and an FID of 0.63. Additionally, we conduct analyses and find that, even for the largest dataset, i.e., HumanML3D, it may still limit the performance of the proposed approach. This suggests that a larger dataset could bring additional improvement to our approach.
Text-to-motion generation must produce motions that are semantically correct, temporally coherent, and physically plausible. A natural approach is to first project motion data into a structured semantic space and then train a generative model within that space. Such a paradigm has been highly successful in image generation through Representation Autoencoders (RAEs), where a frozen self-supervised encoder provides semantic features for diffusion or flow models to learn from. However, direct transfer of such a paradigm to motion space using Motion-JEPA as the frozen encoder fails dramatically. We diagnose this failure geometrically and identify two motion-specific bottlenecks: (1) the JEPA feature space is spectrally ill-conditioned, making the Gaussian-to-data transport unstable; and (2) even with a well-conditioned spectrum, flow residuals tend to align with decoder-sensitive directions, where small latent errors are amplified into large motion artifacts after decoding. Based on these insights, we propose MoRAE. MoRAE addresses the two bottlenecks separately. A compact bottleneck distills the structured JEPA representation while removing weak and redundant directions, bringing the latent spectrum into a transport-stable regime. Motion-coupled training then aligns the retained latent geometry with the decoder, making characteristic flow errors less costly after decoding. With this flow-friendly latent, a standard non-autoregressive Flow-Matching DiT achieves state-of-the-art performance.
Yifei Zhu, Mingyi Shi, Yangyang Cai et al.· 0 citations
Text-driven human motion synthesis has made substantial development with two core modules of motion representation and generative architecture. For representation, Vector Quantization (VQ)-based methods compress motion data into discrete tokens while latent-based models operate directly in continuous space. However, both of these representations exhibit significant limitations. VQ-based methods suffer from inherent information loss, which compromises the quality, diversity, and generalization of generated motions, while continuous representation on holistic whole-body motion hinders part-level flexibility. For architecture, diffusion and autoregressive diffusion models have demonstrated their superiority, yet the fine-grained controllability over individual body parts is also limited. Thus, we propose a unified spatiotemporally decoupled framework named DeMoDiff, which jointly redesigns representation and architecture. To enhance representation extraction capabilities and offer greater part-level controllability, we present a spatial-temporal VAE that encodes each body joint rather than compressing the whole-body motion into a single latent space. Then, we incorporate spatial-temporal masking and attention mechanisms into an autoregressive diffusion generator, achieving both generative capability and controllable editability. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that our model achieves state-of-the-art reconstruction performance and compelling motion generation results. Moreover, our framework demonstrates strong temporal and spatial editing capabilities, further validating its effectiveness. Our project page: https://rex0191.github.io/DeMoDiff/
Chengqun Yang, Liang Xu, Yanping Li et al.· 0 citations
This work proposes FlashMo, a frequency-aware sparse motion diffusion model that prunes low-frequency tokens to enhance efficiency without custom kernel design, and introduces MotionSiT, a scalable diffusion transformer based on a joint-temporal factorized interpolant with Lie group geodesics over SO(3) manifolds, enabling principled generation of joint rotations.
Zeyu Zhang, Yiran Wang, Danning Li et al.· Advances in Neural Informati...· 11 citations
Text-to-motion generation aims to synthesize semantically consistent and naturally coherent motion sequences from natural language descriptions. Given the continuous nature of human motion, diffusion models operating in a continuous latent space offer inherent advantages over vector quantization-based methods, particularly in avoiding quantization errors and in modeling quality. However, existing diffusion models primarily rely on mean squared error loss. This stepwise regression paradigm often leads to ‘over-smoothed’ motion sequences and struggles to capture the subtle semantic nuances embedded in textual descriptions. To realize the potential for continuous diffusion generation, an enhanced latent-space diffusion framework designed to elevate generation capabilities across two dimensions, namely, distribution approximation and semantic alignment, is proposed. Specifically, a latent-space adversarial discriminator is incorporated. By applying decoupled adversarial supervision, this component mitigates the detail loss caused by mean regression, significantly enhancing the physical realism and dynamic sharpness. Concurrently, a latent-space contrastive alignment strategy is introduced during the denoising process that reinforces the correspondence of the generated motion sequences with the given textual inputs via explicit cross-modal constraints. Extensive experiments on standard benchmarks demonstrate that the proposed method effectively addresses the limitations of conventional diffusion models, thus validating the potential of continuous diffusion frameworks within the domain of text-driven motion synthesis.
Zhaowu Li, Rui Liu, Deheng Zhu et al.· Visual Computing for Industr...· 0 citations
Text-to-image generation is an increasingly fast-paced field of generative artificial intelligence, consisting of synthesizing images of high quality and semantic consistency based on natural language descriptions. In this paper, we give an extensive overview of the approach to text-to-image generation using deep learning, including the most common core model families, architecture designs, training approaches, and evaluation systems. We discuss the paradigms of the generative adversarial networks (GANs), variational autoencoders (VAEs), transformer-based designs, and diffusion models, with the last one representing the state of the art in image generation models. The review also discusses key aspects of pipelines such as text encoding, cross-modal alignment, mechanisms of attention, and decoding images. Popular datasets, methods, and metrics of evaluation, including Fréchet Inception Distance (FID) and CLIP-based similarity, are discussed. The application domains that involve creative content creation, medical imaging, education and industrial design are critically discussed. Despite significant advances, various issues still exist, such as low stability in training, excessive computational complexity, amplification of bias, generated images, and text–image alignment errors. Moral and social issues, such as misinformation, intellectual property, and equity, are critically examined. Lastly, we present future research directions to more controllable, more efficient and more interpretable text-to-image systems, focusing on multimodal foundation models and human–AI collaborative design.
GenCeption is introduced, which leverages a pre-trained video generative diffusion backbone to define a feed-forward perception model, capable of performing various vision tasks steered by text instructions, and suggests that video generation is not merely a synthesis tool, but a foundational path toward generalist vision intelligence for the physical world.
Letian Wang, Chuhan Zhang, Rishabh Kabra et al.· 6 citations
