How Much Gradient Does Truncated Backprop Lose in Looped Transformers?
This work measures how much of the weight gradient this throws away, exactly and for every k from one backward pass, on nine looped language models, and reports three findings that show a trained model's value overstates what truncation removes later.