Aquavit: Ascending Quantization for Communication-Efficient Vast-Scale Distributed Training

Abstract

Training Large Foundation Models (LFMs), including Large Language Models and Vision-Language Models, on massive distributed GPU clusters is increasingly bottlenecked by communication overhead. While frameworks like ZeRO++ employ static quantization to reduce communication volume, they suffer from a rigid trade-off: aggressive low-bit quantization maximizes throughput but introduces a persistent noise floor that impedes late-stage convergence, whereas high-precision quantization preserves accuracy at the cost of diminished training efficiency. In this paper, we propose the Noise Equivalent Hypothesis (Neophyte): the noise inherent to the weight quantization process plays a role functionally analogous to the stochastic gradient variance present in standard SGD optimization. We theoretically prove that just as a decaying learning rate is essential for convergence, an ascending quantization schedule is necessary to maintain optimization stability while maximizing throughput. Building on this hypothesis, we present Aquavit, an adaptive distributed training framework that implements an ascending precision schedule for model weights. To overcome the hardware limitations of discrete bit-widths (e.g., jumping from 4-bit to 6-bit), Aquavit introduces fractional precision via granularity modulation. By dynamically adjusting quantization block sizes, we achieve quasi-continuous transitions in effective bits-per-weight (bpw), preventing ''numerical shocks'' and ensuring a smooth error-reduction curve. We evaluate Aquavit on a 256-GPU cluster across diverse LFMs, including LLaMA-3.3-70B and Qwen-Image. Our results demonstrate that Aquavit achieves up to a 2.15× speedup in time-to-accuracy while maintaining full numerical parity with FP16 baselines, even in sensitive multimodal tasks where static quantization typically fails.