Skip to content

Loop Dropout: Regularizing Shared Updates in Looped Language Models

Sep 2026 · 0 citations · 79 references
Computer Science

TL;DR

Loop Dropout is introduced, which couples stochastic masking of adapter applications with inverse-survival rescaling to preserve expected update strength and promote effective adaptation across loops and outperforms existing LoRA variants and adapter regularizers.

Abstract

Looped language models separate computational depth from parameter count by repeatedly applying the same transformer block. Adapting these models requires a shared update that remains effective as hidden states evolve throughout the recurrent computation. Our empirical analysis reveals a pronounced late-loop bias in standard low-rank adaptation (LoRA): the shared update provides limited adaptation at early loop positions. This imbalance motivates training shared updates under varying combinations of their applications. Randomly omitting adapter applications alone, however, does not improve task performance; it reduces expected update strength during training while leaving inference unchanged. We introduce Loop Dropout, which couples stochastic masking of adapter applications with inverse-survival rescaling to preserve expected update strength and promote effective adaptation across loops. Extensive experiments demonstrate improved mathematical reasoning across model sizes, adapter ranks and training recipes, with benefits extending to general instruction tuning and code generation. Loop Dropout outperforms existing LoRA variants and adapter regularizers, while further analysis shows stronger early-loop adaptation. Every backbone loop remains active, and inference applies the adapter at all loops using standard LoRA without additional trainable parameters or inference computation. Code is available at https://github.com/NUS-HPC-AI-Lab/loop-dropout .

View source

Similar papers

Preprint Aug 2026

Allocating Recurrent Compute in Looped Language Models

Results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN, and view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout.

Ruhai Lin, Yi-Yang Guo, Ruiming Zhu et al. · 4 citations
#machine learning Preprint Oct 2026

Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models

Looped Language Models (LoopLMs) offer a parameter efficient approach to scaling reasoning by reusing shared parameters across recurrent computation steps. Despite their promise, effective post-training of LoopLMs remains challenging. Existing approaches either provide reward based supervision that is sparse or costly...

Yi Wang, Rui Qian, Yu Li et al. · 0 citations
#machine learning Preprint Sep 2026

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

RecurTrace introduces Loop Memory Attention, which lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone.

Yu-Xiang Wang, Kun-Yu Feng, Ying-Da Shen et al. · 7 citations
#artificial intelligence Preprint Oct 2026

Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

In this paper, we argue that Looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We first introduce an interesting phenomenon we observed in Looped Transformers, called latent overthinking: noisy state updates overwrite correct intermediate deduc...

Peng-Xiang Li, Dilxat Muhtar, Di He et al. · 1 citation

What Makes Recurrence Effective in Looped Language Models?

Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness. Through controlled experiments,...

Xin-Lin Zhuang, Si-Yuan Wang, Imran Razzak et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts

Looped Transformers introduce recurrent depth as a new scaling axis for LLMs: by repeatedly applying shared Transformer blocks, they increase effective depth without increasing parameter count. However, the benefits of looping remain unclear for large MoE LLMs under FLOPs-matched comparisons. The main reason is that th...

Di He, Peng-Xiang Li, Da Chang et al. · 3 citations

Related blog posts

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.