Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks
Stiefel-AdamW is introduced, a near drop-in replacement for AdamW for use wherever such blocks appear, and it is shown that the resulting optimizer inherits both the stability benefits of Riemannian methods and standard convergence guarantees.
Emanuele Zangrando, M. Sutti, Francesco Tudisco
· 0 citations