Tunneling the Loss Landscape: Bypassing Memorization with Monte Carlo Parameter Swapping
Grokking is a striking phenomenon in neural network training, where a model can undergo a prolonged period of pure memorization before abrupt generalization. While previous works have attempted to interpret it through classical machine learning mechanisms like weight norm, recent research draws an analogy from statisti...