Temporal Self-Conditioning Diffusion Language Model (TDLM) is introduced, which lets each unresolved position attend to its own history of hidden representations within a frozen denoiser, which gives later predictions access to computation already performed.
Abstract
Masked diffusion language models repeatedly compute representations for unresolved tokens, but conventional denoisers do not explicitly retain these representations for reuse across steps. We introduce Temporal Self-Conditioning Diffusion Language Model (TDLM), which lets each unresolved position attend to its own history of hidden representations within a frozen denoiser. This gives later predictions access to computation already performed. Training uses self-generated trajectories and a simple cross-entropy objective for tokens revealed between consecutively sampled decoding steps, while causal temporal attention permits parallel evaluation during training. On LLaDA, the temporal modules add only approximately 0.84% trainable parameters. Across LLaDA and Dream, the main comparison shows up to 7.15$\times$ faster generation than the corresponding original diffusion language model, with competitive answer quality on the evaluated tasks. These results motivate temporal memory as a practical way to make more effective use of repeated denoising computation.
ALoDLM is introduced, which replaces uniform computation with token-adaptive latent recurrence, and retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.
Lian-Cheng Fang, Zhuo-Wei Li, Youngeun Kim et al.· 1 citation
Diffusion language models reconstruct masked tokens in parallel, offering an alternative to autoregressive generation. Yet their Transformer backbones typically carry visible evidence and evolving masked predictions through a single residual stream. We equip masked diffusion models with multiple residual streams that p...
Ekaterina Alimaskina, S. Kazarian, G. Molodtsov et al.· 0 citations
We introduce a post-training method for diffusion language models (DLMs) that minimizes Maximum Mean Discrepancy (MMD) between generated and reference distributions in the feature space of a frozen pretrained DLM. To estimate MMD, we retain contextual features at individual token positions, obtaining multiple observati...
Ilya Drobyshevskiy, I. Sudakov, Maksim Semenov et al.· 0 citations
PUMBA is introduced, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time.
Manuel Madeira, Amitis Shidani, Alice Bizeul et al.· arXiv.org· 2 citations· ⚡1
This work introduces the Hierarchical Masked Diffusion Language Model (HMDLM), which augments each token position with a hierarchy level available as part of the training data, and requires zero or minimal additional annotations.
Parul Gupta, Satwik, Avni Aggarwal et al.· 0 citations
This design introduces a rewritable proposal that uses additional compute to improve expressivity while keeping the simple mask-based transition schedule of MDMs, and fine-tune Qwen2.5-7B and Qwen3-8B to obtain LED-\texttt{7B} and LED-\texttt{8B}, respectively.
Yu-Yan Zhou, Ze-Yuan Yang, Yuxuan Song et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.