Preprint Open access
Language-Specific Effects of Tokenizer Choice in Multilingual Language Models
Tokenizer choice affects multilingual language modeling, but vocabulary capacity is finite and vocabulary size is often constrained: improving representation for some languages often comes at the expense of others. We therefore ask whether tokenizer choice matters equally across languages, a question that the current l …