Open access
2026
Evaluating Multilingual Tokenization under Worst-N Parity-Aware BPE
This paper presents a multilingual evaluation of parity-aware to-kenization under worst-N optimization, extending PA-BPE to jointly optimize over the N worst-compressed languages.
Vani Kanjirangat, David Kletz, Tanja Samardzic et al.
· Proceedings of the 1st Works... · 0 citations