Stabilizing the Dynamic Low-Rank Training
Training neural networks directly in a low-rank parameterization is an appealing route to reducing memory, compute, and storage simultaneously during both training and inference. Dynamic low-rank training (DLRT), which confines weights to a rank-$r$ manifold via the Galerkin projection of the gradient flow, is particul...
Zhong-Han Xu, Ling Wang, Jun-Hao Chen et al.
· 0 citations