Aquavit: Ascending Quantization for Communication-Efficient Vast-Scale Distributed Training
The Noise Equivalent Hypothesis is proposed: the noise inherent to the weight quantization process plays a role functionally analogous to the stochastic gradient variance present in standard SGD optimization, and it is theoretically prove that just as a decaying learning rate is essential for convergence, an ascending...