Language-Specific Effects of Tokenizer Choice in Multilingual Language Models
It is found that tokenizer choice matters more for languages with less language-model training data: across the 54 tokenizers, the standard deviation of a language's bits-per-byte (BPB) increases as its model training-data share decreases.
Clara Meister, G. S. Altintas, Antoine Bosselut
· 0 citations