No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance
Tokenization is a critical component of the language modeling pipeline, but it is poorly understood. Tokenizers can impact downstream performance, but there is currently mixed evidence about which tokenizer features predict a better language model without requiring a full training run. In this paper, we analyze approxi...
Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava et al.
· 0 citations