Muhammed Ökmen

ZeroFloat: Fully Integer Training of Language Models Matches AdamW up to 1B Parameters

We train transformer language models from scratch in which every training operation is integer arithmetic: the forward pass (matrix products, RMSNorm with integer square root, multi-head attention with an integer base-2 softmax), the backward pass (including the softmax and normalization Jacobians), and the parameter u...