ZeroFloat: Fully Integer Training of Language Models Matches AdamW up to 1B Parameters

Abstract

We train transformer language models from scratch in which every training operation is integer arithmetic: the forward pass (matrix products, RMSNorm with integer square root, multi-head attention with an integer base-2 softmax), the backward pass (including the softmax and normalization Jacobians), and the parameter update. Weights are stored as int8 with a per-tensor power-of-two scale, there is no floating-point master copy, and the optimizer is an integer, Adafactor-like rule: log2-quantized gradient magnitudes, a factored (row x column) second-moment estimate kept in integers, and an integer accumulator that moves a weight by one int8 step when it crosses a threshold. On character-level language modeling with a fixed budget, ZeroFloat matches a tuned AdamW baseline at every size we tested, from 0.8M to 1.0B parameters (gap in bits/char: -0.061, -0.064, -0.007, +0.020, +0.008). A single size rule for the threshold, fitted on smaller models, predicted the best 1B setting in advance. At 1B, peak training memory is 11.2 GB vs. 20.2 GB for AdamW. We also report a failure mode found on the way (silent int16 accumulator overflow at large thresholds), which alone made the 1B model appear 0.24 bits/char worse than AdamW. Version 1: 1B result from a single seed; additional seeds and a long-horizon study will be added in later versions.