Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal
The edge-of-stability (EoS) phenomenon of full-batch Adam has been widely observed, yet its underlying dynamical mechanism remains poorly understood. In this paper, we identify Adam's second-moment adaptation as a negative-feedback mechanism that drives the dynamics toward the stability boundary. We characterize this m...
Yiman Fong, Heng Yang
· 1 citation