How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference
The results show that effective looping requires matching the design to the resource constraint and inference setting, and that preserving the current-token embedding and separately normalizing it and the previous-token hidden state before combining them recovers strong performance across model scales and training budg...
C. Ekbote, Artidoro Pagnoni, Devendra Singh Sachan et al.
· 0 citations