A Theory of Saddle Escape in Deep Nonlinear Networks
Abstract
In deep networks with small initialization, training can exhibit long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive exact scalar and matrix identities for the imbalance of layer weight norms, holding for any smooth activation and any differentiable loss, and use the resulting approximate balance law to control the full finite-width gradient flow through its first escape. This gives a scaling law in the escape time: with a prefix of $r$ layers initialized at the bottleneck scale $\varepsilon$, the escape time is $\Theta(\Gamma^{-1})$ for $r=1$, $\Theta(\Gamma^{-1}\log(1/\varepsilon))$ for $r=2$, and $\Theta(\Gamma^{-1}\varepsilon^{-(r-2)})$ for $r\geq3$, where $\Gamma=\|\mathbb{E}[yx]\|$. We further show that a multimode teacher leaves the saddle once rather than in stages: with scalar output, all mode overlaps grow through a common amplitude, with relative weights fixed in advance. We find close agreement between our theory and experiment.
