Treating the mid-twentieth-century simplification reform as a controlled compression event, this work finds it raised optimality from 0.555 to 0.668, with savings concentrated in the 1,000 commonest characters.
Abstract
Zipf's law of abbreviation -- the tendency of frequent forms to be short -- is one of the best-supported regularities in language, and recent work has moved from demonstrating it to measuring how far lexicons are compressed relative to principled baselines. That programme has so far addressed word lengths in alphabetic and syllabic scripts. We transfer it to a logographic script, taking the stroke as the unit of articulatory cost and the Chinese character as the coded form. Combining a stroke-order database covering all 20,902 characters of the CJK basic block with two independent frequency corpora (258.9M and 193.3M tokens), we find that the mean character type costs 12.71 strokes but the mean character token in running text only 7.22. Using the dually normalised optimality score of Petrini et al. (2026), the simplified inventory reaches Omega = 0.668, with the replication corpus at 0.609 -- inside and just below the 62-67% band those authors report for word lengths across 20 languages and 8 scripts, suggesting a compression ceiling largely independent of script type and cost unit. A logographic script also makes absolute coding bounds computable, since strokes come from a closed five-element taxonomy: the exact 5-ary Huffman optimum is 4.34 strokes and the entropy bound 4.28, so the observed system is 1.66x above optimal coding. This gap is not slack but structure. The Kraft sum of 5^(-l_i) is 2.05 on the frequency list and 5.03 on the full inventory, so stroke strings are provably not uniquely decodable in one dimension; characters are disambiguated by the two-dimensional arrangement of strokes, not their sequence, and the forgone compression buys componential transparency. Finally, treating the mid-twentieth-century simplification reform as a controlled compression event, we find it raised optimality from 0.555 to 0.668, with savings concentrated in the 1,000 commonest characters.
A deterministic, LLM-free Quranic recitation validator using a four-layer verse-matching search (exact, morphological, relaxed, fuzzy) and word-error-rate-graded feedback across five severity tiers is built.
Downstream the effect is small: NER improves only on entities that contain words unseen in training, POS tagging and news classification do not change, and published Nepali tokenizers perform about as well.
The classical formulation of Heaps' Law establishes a bivariate relationship between the length of a text and the growth of its vocabulary, disregarding morphological, phonological, and stylistic factors inherent to natural language. This work proposes and validates a compact, generalized extension of Heaps' Law, incor...
R. Linhares, Regis Nunes Vargas· Revista Ibero-Americana de H...· 0 citations
Tokenization is the first design choice in a language model and one of the most frequently overlooked for morphologically rich languages, as it directly shapes sequence length and how related word forms are segmented into reusable sub word units. Using Russian as a case study, we train sub word tokenizers from scratch...
Byte-level BPE tokenizers that use the HuggingFace ByteLevel pre-tokenizer inherit GPT-2's word regex, where a word is defined as \p{L}+, one or more Unicode letters, which splits each word at every vowel sign, and formalises this effect as a training-free lower bound on fertility.
S. Regmi, Siddhartha Pudasaini, Chetan Phakami Pun· 3 citations
Large language models (LLMs) do not read words; they read tokens, and the tokenizer decides how many tokens a language needs to say the same thing. This paper introduces the notion of a Sindhi token tax and tests it empirically for Pakistan’s English–Sindhi context. Using 1,004 parallel sentences from the FLORES-200 be...
Ali Siddiqui, Shauban Ali Solangi, Uzma Ehsan· Journal of Social Research D...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.