Ori: Drift-Aware Streaming $k$-PCA for Very Low-Rank LLM Training
A prominent family of low-rank LLM optimizers, pioneered by GaLore, performs gradient-subspace projection: at each step, they project the gradient into a low-dimensional subspace and map the update back to the full model. Most existing work either recomputes the gradient subspace at certain intervals, or performs post-...
Xin-Yu Luo, Jin-Cheng Zhou, Meng-Bo Wang et al.
· 0 citations