Gradient Prediction with Control Variates in the Cheap-Forward Regime
On a 124M-parameter language model and selected short training windows, the method can lower simulated ledger cost relative to the tested baselines when fleet work is sufficiently cheap, and can lower simulated ledger cost relative to the tested baselines when fleet work is sufficiently cheap.
K. Ciosek, Nicolò Felicioni, Juan Elenter et al.
· 2 citations