Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
Taken a step further, theoretical analysis is established that characterizes how momentum accelerates optimization by stabilizing large learning rates that can not be tolerated by vanilla GD without deviating significantly from the river.