Looped large language models trade parameter storage for recurrent computation, but most are trained from scratch. In practice, pretraining with looped depth is expensive, making post-training upcycling of a pretrained non-looped LLM an attractive alternative. We introduce \textbf{UpLoop}, a post-training recipe based...
Tie-Yuan Chen, Hua-Bin Liu, Ying-Tao Zhang et al.· 0 citations
Speculative decoding accelerates language model inference by verifying multiple draft tokens in a single target-model pass. Recent parallel drafters have achieved breakthrough performance in frontier production models, but their effectiveness deteriorates as the entropy of target distributions increases due to insuffic...
Prefix-Conditioned Diffusion (PCD), which samples a boundary, preserves the prefix, preserves the prefix, and denoises the suffix, is proposed, and improvements in reasoning and coding over native diffusion training are shown.
Xiao-Cheng Lu, Hua-Bin Liu, Song Guo et al.· 0 citations
This work systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models.
Fengqi Zhu, Shaoxuan Xu, Jingyang Ou et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.