Ping Luo

Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

This work jointly varies model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters, showing that at fixed model size, the compute-optimal loop count and training-token b...