Skip to content
Preprint

Hierarchical Masked Diffusion Language Models

Oct 2026 · 0 citations · 28 references
Computer Science

TL;DR

This work introduces the Hierarchical Masked Diffusion Language Model (HMDLM), which augments each token position with a hierarchy level available as part of the training data, and requires zero or minimal additional annotations.

Abstract

Masked diffusion language models have emerged as an alternative to autoregressive models, enabling parallel token generation. However, standard masked diffusion language models use a shared noise schedule across positions without explicitly encoding which positions should be resolved first. Many tasks exhibit a natural priority among positions: document titles and section headings can guide the generation of body text, while Sudoku cells determined directly from the initial clues can help resolve more difficult cells. To incorporate this notion of priority (or hierarchy) among token positions, we introduce the Hierarchical Masked Diffusion Language Model (HMDLM), which augments each token position with a hierarchy level available as part of the training data, and requires zero or minimal additional annotations. It factors generation into two passes: the first predicts a hierarchy distribution over masked positions, the second predicts token distributions conditioned on it. Giving each hierarchy level its own noise schedule induces a soft, controllable preference for revealing higher-priority positions earlier while keeping decoding fully parallel. Our formulation is mathematically grounded, and we re-derive the forward and reverse processes and the NELBO for this factorization. Our evaluation shows that for the task of language generation on LM1B, HMDLM reduces perplexity by more than 1.5 points over Anchored Diffusion Language Model (ADLM), a recently proposed strong diffusion baseline; on WikiHow, we see a drop in perplexity of up to 4.4 pts on ADLM and up to 7 pts against MDLM. An LLM judge on WikiHow prefers HMDLM generations (the two variants combined) more than $2\times$ as often as the baseline. On the task of solving Sudoku puzzles requiring multi-step reasoning, it improves solution accuracy by up to $8.6\%$ absolute points over existing diffusion baselines.

View source

Similar papers

#machine learning Preprint Oct 2026

Multi-Stream Residual Computation in Diffusion Language Models

Diffusion language models reconstruct masked tokens in parallel, offering an alternative to autoregressive generation. Yet their Transformer backbones typically carry visible evidence and evolving masked predictions through a single residual stream. We equip masked diffusion models with multiple residual streams that p...

Ekaterina Alimaskina, S. Kazarian, G. Molodtsov et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Hierarchical Continuous Diffusion Language Models

H Hierarchical Continuous Diffusion Language Models (HC-DLM) is proposed, which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood.

Hui Ren, Zi-Han Li, Chang Liu et al. · 0 citations
#machine learning Preprint Oct 2026

LED: Improving Masked Diffusion Language Models by Lookahead Editing

This design introduces a rewritable proposal that uses additional compute to improve expressivity while keeping the simple mask-based transition schedule of MDMs, and fine-tune Qwen2.5-7B and Qwen3-8B to obtain LED-\texttt{7B} and LED-\texttt{8B}, respectively.

Yu-Yan Zhou, Ze-Yuan Yang, Yuxuan Song et al. · 0 citations

Zarya: A Hybrid Autoregressive-Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference

Zarya is introduced, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective within a single architecture, demonstrating performance on standard benchmarks while offering a principled integration of autoregressive and diffusion paradigms.

Leonid S. Sinev, Ilya Koziev, Vladislav Leshchuk · 0 citations
Preprint Oct 2026

TDLM: Adaptive Temporal Integration for Diffusion Language Models

Temporal Self-Conditioning Diffusion Language Model (TDLM) is introduced, which lets each unresolved position attend to its own history of hidden representations within a frozen denoiser, which gives later predictions access to computation already performed.

Bo-Han Sun, Max Zhu, Francesco Caso et al. · 0 citations
#artificial intelligence Preprint Oct 2026

ALoDLM: Adaptively Looped Diffusion Language Models

ALoDLM is introduced, which replaces uniform computation with token-adaptive latent recurrence, and retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.

Lian-Cheng Fang, Zhuo-Wei Li, Youngeun Kim et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.