This work proposes Parallel Autoregressive Decoding (PARD), a simple training-free sampling method that preserves left-to-right unmasking structure while allowing parallel token commitment and consistently outperforms existing parallel samplers in generation quality.
Abstract
Diffusion language models enable flexible arbitrary-order generation, but existing sampling methods are mostly designed for early masked diffusion models (MDMs). In this work, we study sampling for recent block diffusion language models (BDLMs). We show empirically and analytically that these models are naturally more aligned with left-to-right decoding than MDMs. Based on this observation, we propose Parallel Autoregressive Decoding (PARD), a simple training-free sampling method that preserves left-to-right unmasking structure while allowing parallel token commitment. Extensive experiments show that PARD consistently outperforms existing parallel samplers in generation quality, while achieving substantial speedups over pure AR decoding with only a small quality gap.
Zarya is introduced, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective within a single architecture, demonstrating performance on standard benchmarks while offering a principled integration of autoregressive and diffusion paradigms.
Leonid S. Sinev, Ilya Koziev, Vladislav Leshchuk· 0 citations
This work develops a trajectory-level speculative framework that constructs draft denoising trajectories via confidence-stratified tree exploration and verifies them through blockwise parallel evaluation with bidirectional attention masking, and introduces inter-block speculation, exploiting diffusion models'bidirectio...
Tian-Xiang Pan, Baitao Gong, Mo Guang et al.· 0 citations
Across reasoning and code-generation benchmarks, survival-guided length decoding speeds up inference by up to 7 times while preserving task accuracy, and it is found that predicted lengths vary widely even within the same dataset, making model performance sensitive to the chosen length.
I. Kobyzev, Abbas Ghaddar, Yu-Fei Cui· 0 citations
Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation. To overcome this bottleneck, we introduce diffusion-augmented LLMs, a new class of models that defines an AR model distribution while using diffusion...
S. Sahoo, Ling-Jie Chen, Khiem Pham et al.· 1 citation
Temporal Self-Distillation (TSD), a simple on-policy method that trains dLLMs for fast inference by distilling predictions across time, substantially shifts the speed--quality frontier toward the low-compute regime.
Shi-Jia Xu, Andrea Miele, Metod Jazbec et al.· 0 citations
Spherical Soft-Masking (S-SM), a drop-in replacement that aggregates the top-(k) predictions with a Fr'echet mean on the hypersphere and blends this mean with the mask direction using spherical linear interpolation (SLERP), then restores the native mask norm.
Lavanya Nigam, Ishaan Bansal, Aryan Sood et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
