Skip to content

Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models

Sep 2026 · Conference on Empirical Methods in Natural Language Processing · Vol abs/2609.25927 · 0 citations · 20 references
Computer Science

TL;DR

This work identifies a systematic upstream/downstream structure in dLLM rollouts and proposes Informed Masking (IM), which derives a per-token priority score from the denoising trajectory at zero extra inference cost and biases mask sampling toward downstream tokens.

Abstract

Diffusion Large Language Models (dLLMs) have emerged as an efficient alternative to autoregressive models, yet aligning them via Reinforcement Learning (RL) requires likelihood surrogates estimated from masked reconstruction subproblems under a small Monte Carlo budget per rollout. Existing methods construct these subproblems by uniform random masking, leaving open the question of which subproblems to prioritize. We identify a systematic upstream/downstream structure in dLLM rollouts. Some tokens, when revealed, trigger large confidence changes in nearby undecoded positions; we call them upstream. Others induce only small local changes and are therefore downstream. We find masking downstream tokens yields substantially better-posed subproblems than masking upstream tokens, a phenomenon we term subproblem difficulty asymmetry. Based on the observation, we propose Informed Masking (IM), which derives a per-token priority score from the denoising trajectory at zero extra inference cost and biases mask sampling toward downstream tokens. IM is plug-and-play: when plugged into three state-of-the-art dLLM RL methods on LLaDA-8B-Instruct, it delivers up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks with improved training stability.

Read PDF

Similar papers

#artificial intelligence Preprint Oct 2026

ALoDLM: Adaptively Looped Diffusion Language Models

ALoDLM is introduced, which replaces uniform computation with token-adaptive latent recurrence, and retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.

Lian-Cheng Fang, Zhuo-Wei Li, Youngeun Kim et al. · 1 citation

On Trajectory-Aware Training for Masked Diffusion Language Models

PUMBA is introduced, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time.

Manuel Madeira, Amitis Shidani, Alice Bizeul et al. · 2 citations · ⚡1
#machine learning Preprint Oct 2026

IDRF: Inverse-Distilled Reward Fine-tuning of Masked Discrete Diffusion Models

IDRF, a framework for reward fine-tuning of few-step masked discrete diffusion generators, view few-step generation as a finite-horizon Markov decision process and optimize reward with a clipped policy-gradient objective over the student's trajectories.

V. Gromadskii, David Li, Samson Gourevitch et al. · 1 citation
#artificial intelligence Preprint Sep 2026

Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Models

This work proposes stepwise risk-sensitive GRPO (StepRS-GRPO), which varies the risk coefficient of the group-advantage transformation across denoising states while retaining the underlying trainer, and shows that the diversity gains from StepRS-GRPO extend beyond final-answer strings.

Yue Yu, Bo-Wen Zuo, David J. Crandall et al. · 1 citation

Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models

Masked diffusion language models (dLLMs) have shown strong potential for faster inference through parallel token generation when combined with confidence-based samplers. However, recent work has shown that such methods can defer unmasking high-entropy fork positions at which multiple plausible continuations exist. This...

S. Frkovic, Metod Jazbec, C. Naesseth · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.