Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models
This work takes a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters, finding that memorization in both cases is characterized by sequence-level gradient alignment.
Florian Eichin, Philipp Mondorf, Andrei Mircea et al.
· 0 citations