Д. Л. Тихонов

Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining

Training large language models is constrained by both wall-clock time and memory footprint, and recent matrix-aware optimizers sharpen this tradeoff: Muon can improve optimization but its dense polar update remains expensive. We propose Monarch-Muon, a structured matrix-aware optimizer for weights parameterized by Mona...