Sep 2026· Conference on Empirical Methods in Natural Language Processing· Vol abs/2609.25927· 0 citations· 20 references
Computer Science
TL;DR
This work identifies a systematic upstream/downstream structure in dLLM rollouts and proposes Informed Masking (IM), which derives a per-token priority score from the denoising trajectory at zero extra inference cost and biases mask sampling toward downstream tokens.
Abstract
Diffusion Large Language Models (dLLMs) have emerged as an efficient alternative to autoregressive models, yet aligning them via Reinforcement Learning (RL) requires likelihood surrogates estimated from masked reconstruction subproblems under a small Monte Carlo budget per rollout. Existing methods construct these subproblems by uniform random masking, leaving open the question of which subproblems to prioritize. We identify a systematic upstream/downstream structure in dLLM rollouts. Some tokens, when revealed, trigger large confidence changes in nearby undecoded positions; we call them upstream. Others induce only small local changes and are therefore downstream. We find masking downstream tokens yields substantially better-posed subproblems than masking upstream tokens, a phenomenon we term subproblem difficulty asymmetry. Based on the observation, we propose Informed Masking (IM), which derives a per-token priority score from the denoising trajectory at zero extra inference cost and biases mask sampling toward downstream tokens. IM is plug-and-play: when plugged into three state-of-the-art dLLM RL methods on LLaDA-8B-Instruct, it delivers up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks with improved training stability.
The findings identify the denoising trajectory as a new control channel in dLLMs and call for bias audits that examine the serving stack rather than the frozen model alone.
Sarim Hashmi, Mukul Ranjan, Abdelrahman W. A. Elsayed et al.· 0 citations
ALoDLM is introduced, which replaces uniform computation with token-adaptive latent recurrence, and retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.
Lian-Cheng Fang, Zhuo-Wei Li, Youngeun Kim et al.· 1 citation
PUMBA is introduced, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time.
Manuel Madeira, Amitis Shidani, Alice Bizeul et al.· arXiv.org· 2 citations· ⚡1
IDRF, a framework for reward fine-tuning of few-step masked discrete diffusion generators, view few-step generation as a finite-horizon Markov decision process and optimize reward with a clipped policy-gradient objective over the student's trajectories.
V. Gromadskii, David Li, Samson Gourevitch et al.· 1 citation
This work proposes stepwise risk-sensitive GRPO (StepRS-GRPO), which varies the risk coefficient of the group-advantage transformation across denoising states while retaining the underlying trainer, and shows that the diversity gains from StepRS-GRPO extend beyond final-answer strings.
Yue Yu, Bo-Wen Zuo, David J. Crandall et al.· 1 citation
Masked diffusion language models (dLLMs) have shown strong potential for faster inference through parallel token generation when combined with confidence-based samplers. However, recent work has shown that such methods can defer unmasking high-entropy fork positions at which multiple plausible continuations exist. This...
S. Frkovic, Metod Jazbec, C. Naesseth· arXiv.org· 0 citations