Skip to content

FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

Sep 2026 · 0 citations · 27 references
Computer Science

TL;DR

FlashLoop is introduced, a training-free inference framework that reduces cross-loop redundancy through token-sparse updates, sparse attention, and KV-residual quantization, substantially improving the practicality of scaling Looped Transformers to greater computational depths and longer context.

Abstract

Looped Transformers have attracted substantial attention as a parameter-efficient approach to increasing computational depth through repeated application of shared Transformer blocks. However, their practical advantages over conventional Transformers remain under debate: each additional loop incurs another Transformer pass and requires caching another set of KV states, causing inference FLOPs and KV-cache memory to grow continuously with loop depth. This overhead becomes particularly severe at large loop counts and long context, preventing the parameter efficiency of Looped Transformers from translating into practical inference efficiency. In this paper, we find that much of the additional computation and storage introduced by looping is redundant. As recurrence proceeds, state changes become increasingly concentrated on a small subset of tokens; attention-output differences are dominated by a sparse and stable subset of key columns; and KV residuals between adjacent loops become progressively more amenable to low-bit quantization. Building on these observations, we introduce FlashLoop, a training-free inference framework that reduces cross-loop redundancy through token-sparse updates, sparse attention, and KV-residual quantization. Across several Looped Transformers models, FlashLoop delivers lossless accuracy while achieving up to 1.64$\times$ end-to-end speedup and up to 6$\times$ KV-cache memory reduction, substantially improving the practicality of scaling Looped Transformers to greater computational depths and longer context.

View source

Similar papers

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

This work proposes LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers that introduces a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes.

SangLyul Cho, Lang-Qing Cui, Sehoon Kim et al. · 7 citations · ⚡1
#artificial intelligence Preprint Sep 2026

T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning with Dynamic Routing

This work proposes dynamic token-choice routing for looped transformers, enabling each token to adaptively determine its own number of loop iterations based on its hidden state, which can improve the token generation accuracy and validate the effectiveness of token-choice router and recursion-wise KV cache.

Ming-Qian Yu, Wen-Peng Zhang, Shao-Bo Cui et al. · 2 citations

Looped Transformers as Optimizers

This framework derives loop transitions in closed form from a projection, a local objective and an optimizer update rule, which combines explicit weight decay, adaptive step size and a delta objective and improves average generative performance over the compared looped and non-looped baselines under matched training FL...

Yu-Long Huang, Chen-Yu Jiang, Zhan-Peng Zhou et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers

Looped Transformers reuse a shared block across recurrent depths, making autoregressive decoding expensive because every generated token requires many sequential recurrent passes. Self-speculative decoders reduce this cost by drafting at an early depth and verifying at full depth, but typically bind draft computation t...

Guang-Hao Li, Zi-Han Su, Hao Yu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts

Looped Transformers introduce recurrent depth as a new scaling axis for LLMs: by repeatedly applying shared Transformer blocks, they increase effective depth without increasing parameter count. However, the benefits of looping remain unclear for large MoE LLMs under FLOPs-matched comparisons. The main reason is that th...

Di He, Peng-Xiang Li, Da Chang et al. · 3 citations
#machine learning Preprint Oct 2026

Decoding Looped Transformers Better for (Almost) Free

Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies...

Weihao Liu, Huangjie Zheng, Tian-Rong Chen et al. · 2 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.