Skip to content

Author

Zhibek Sarsen

2 papers indexed here

We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.

Not the right person? Other researchers publish under this name.

#small language model Open access Sep 2026

Tokenization Disparity in Kazakh: Cross-Tokenizer Measurement and Low-Cost Vocabulary Adaptation for Large Language Models

Large language models are billed by the token, and tokens are not distributed evenly across languages. We measure how many tokens ten widely deployed tokenizers require to encode identical content in Kazakh and English, using 900 sentence-aligned pairs from OPUS-100 with bootstrap confidence intervals over 10,000 resam...

Alikhan Akim, Zhibek Sarsen · 0 citations
#large language models Open access Sep 2026

Tokenization Disparity in Kazakh: Cross-Tokenizer Measurement and Low-Cost Vocabulary Adaptation for Large Language Models

Large language models are billed by the token, and tokens are not distributed evenly across languages. We measure how many tokens ten widely deployed tokenizers require to encode identical content in Kazakh and English, using 900 sentence-aligned pairs from OPUS-100 with bootstrap confidence intervals over 10,000 resam...

Alikhan Akim, Zhibek Sarsen · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.