Diffusion language models are a promising and more efficient alternative to autoregressive models. However, most RL post-training techniques are not easily adapted to these models. One issue is that diffusion language models generate answers through many intermediate states, deciding at each step which masked positions...
This work pretraining three 3B block-diffusion denoisers (attention, mamba, and hybrid) on 300B tokens under one single-frontier objective and decoding all three through a single cached interface finds the Mamba cache delivers 4.3x lower latency, 11x less memory, and 2.6x higher single-stream throughput.
Vaibhav Singh, Pierre-André Noël, Torsten Scholak et al.· 1 citation
ThinkRetrieve is proposed, a test-time scaling framework that augments the reasoning traces of LRMs with dynamically retrieved solved examples at each reasoning step, providing the model with guidance on how to reason rather than merely what facts are relevant.