This work proposes dynamic token-choice routing for looped transformers, enabling each token to adaptively determine its own number of loop iterations based on its hidden state, which can improve the token generation accuracy and validate the effectiveness of token-choice router and recursion-wise KV cache.
Abstract
Looped Transformers have recently demonstrated strong performance in both reasoning and language tasks by reusing a shared set of parameters across multiple iterations, achieving parameter efficiency without sacrificing representational power. Besides, looped Transformers perform inference directly in the latent space (latent reasoning) to reduce the number of tokens consumed during inference, thereby achieving improved sample efficiency. However, these models typically apply a fixed recursion depth uniformly to every token, leading to suboptimal compute allocation and leaving significant efficiency gains on the table. In this work, we propose dynamic token-choice routing for looped transformers, enabling each token to adaptively determine its own number of loop iterations based on its hidden state, which can improve the token generation accuracy. Moreover, we further introduce recursion-wise KV cache, which maintains an independent key-value cache for each recursion loop, this design ensures that tokens at different depths only attend to their corresponding cached states, effectively enabling faster autoregressive decoding. Extensive experiments show that T-LoopFormer achieves robust performance on language modeling and zero-shot reasoning tasks and our model can reach the lowest decoding latency, which validate the effectiveness of token-choice router and recursion-wise KV cache. Code is available at https://github.com/YuMingQian1234/T-LoopFormer
Looped transformers have demonstrated promising parameter efficiency by reusing layers for latent computation. Prior studies compare looped and non-looped models at matched parameters or per-token FLOPs. However, to the best of our knowledge, whether looping improves test-time scaling as outputs grow longer remains und...
This work proposes LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers that introduces a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes.
SangLyul Cho, Lang-Qing Cui, Sehoon Kim et al.· arXiv.org· 7 citations· ⚡1
Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing, by softly aligning the hidden state of loop $t$ with the embedding of the token $t$ steps ahead, while a lightweight gate preserves useful information across iterations.
Behzad Shomali, Markus Frey, D. Berghaus et al.· 3 citations
This framework derives loop transitions in closed form from a projection, a local objective and an optimizer update rule, which combines explicit weight decay, adaptive step size and a delta objective and improves average generative performance over the compared looped and non-looped baselines under matched training FL...
FlashLoop is introduced, a training-free inference framework that reduces cross-loop redundancy through token-sparse updates, sparse attention, and KV-residual quantization, substantially improving the practicality of scaling Looped Transformers to greater computational depths and longer context.
The results suggest that temporal thinking offers a parameter-efficient alternative to increasing physical depth in recurrent Transformers, and that temporal thinking offers a parameter-efficient alternative to increasing physical depth in recurrent Transformers.
Ze-Yi Huang, Xuehai He, Yong Jae Lee et al.· 0 citations