This work jointly varies model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters, showing that at fixed model size, the compute-optimal loop count and training-token b...
Xu-Yuan Xiong, Si-Yu Chen, Zi-Han Shao et al.· 1 citation
Taken a step further, theoretical analysis is established that characterizes how momentum accelerates optimization by stabilizing large learning rates that can not be tolerated by vanilla GD without deviating significantly from the river.
Miao Lu, Ze-Yu Bian, Kai-Yue Wen et al.· 0 citations
INFUSER is introduced, an iterative co-training framework with two co-evolving roles: a Generator that drafts questions and reference golden answers from a pool of unstructured, automatically collected documents, and a Solver that improves by training on them, and DuGRPO, a dual-normalized variant of GRPO, for generato...