Parallelism Strategy Chaining for Fast Training Convergence
This paper proposes CONA, a new training method that introduces online strategy chaining, which ranks candidate strategies during training using a surrogate metric built from compute throughput and gradient statistics, and switches the current strategy to a new strategy with a higher metric.
Mi-Gyung Kang, Changyong Shin, Younghun Go et al.
· 0 citations