Zhi-Chao Wang

Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding

This work theoretically study how stochastic rounding (SR) affects prediction risk as model size, training time, and precision change, and derives how the smallest prediction error decreases with compute, and identifies model sizes and training times that achieve optimal-compute scaling.