Softmax Reparameterization for Output-Head Quantization
Large vocabularies make output heads a substantial inference cost in small language models. We introduce softmax reparameterization, a post-training method that searches over functionally equivalent output heads before quantization. The method subtracts a scalar multiple of the vocabulary-row mean from every output row...
Asim Kadav, Christian Flores, C. Arora et al.
· 0 citations