UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute
UpLoop is a pretraining schedule that uses upcycling to train looped Transformers while exactly matching a standard dense baseline in final parameter count, total training tokens, and training FLOPs, showing improvements in language modeling and downstream performance.