Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry
The dynamics of Stochastic Gradient Descent is studied by modeling the stochastic gradient flow as a percolation process, in which nested architectural symmetries force subnetworks to merge in discrete blocks rather than by single-edge attachment.
Sai Niranjan Ramachandran, Suvrit Sra
· 0 citations