Courtney Paquette

Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models

Intermittent ``spikes''in loss, during the pre-training of Large Language Models (LLMs), remains a mysterious phenomenon both theoretically and empirically. In this paper, we show that simple quadratic models with heavy-tailed, power-law distributed inputs can replicate these spikes. We introduce a theoretical framewor...