Stochastic Optimization Under Power-Law Spectra: Tight Bounds and Shuffling Analysis
The power-law spectral theory is generalized to Stochastic Gradient Descent (SGD), showing that the same spectral exponents govern stochastic dynamics, and a precise analysis of data shuffling is provided, deriving exact constants that prove Single Shuffle is strictly superior to Flip-Flop and IID sampling.
Christopher De Sa, Thomas Dybdahl Ahle, Yaroslav Bulatov et al.
· 0 citations