Type-Driven Tokenization for Brahmic Scripts
Standard tokenizers used in large language models produce malformed text when applied to Brahmic scripts. They are a family of abugidas, writing systems whose consonants carry an inherent vowel that dependent marks can modify. They include Devanagari, Telugu, Tamil, Kannada, and others. The underlying issue is that the...
S. Kapila, Rakshika Bagavathy
· 0 citations