Clean: Second-order LLM Training at Linear Memory Cost via Nystr{\"o}m Sketching
The methods uniquely enable the pre-training of a 13B-parameter model on a single 80GB GPU, providing a scalable, efficient, and accessible approach to large-scale model optimization.
Beheshteh T. Rakhshan, S. Rajabi, Maziar Sargordi et al.
· 0 citations