Generalizing Adam To Manifolds For Efficiently Training Transformers
A new approach is presented that leverages the special structure of the manifolds which are relevant for optimization of neural networks, such as the Stiefel manifold, the symplectic Stiefel manifold and the Grassmann manifold, which makes the generalization of all steps in the Adam optimizer possible.