Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya

Abstract

Multilingual pre-trained language models such as XLM-R perform well for major languages but struggle with low-resource Ge'ez-script languages, largely due to high out-of-vocabulary (OOV) rates and excessive subword fragmentation from Latin-script-centric tokenizers. We introduce VEXMLM, a vocabulary-extended variant of XLM-R targeting Amharic and Tigrinya. We train language-specific SentencePiece tokenizers on curated monolingual corpora, extend XLM-R's vocabulary with 30k Ge'ez-script subwords, and initialize their embeddings via subword averaging. VEXMLM undergoes two-stage training: (1) continued masked language modeling on the curated corpora and (2) supervised fine-tuning on question answering, named entity recognition, and sentiment analysis. VEXMLM substantially outperforms XLM-R and Glot500 across all evaluated tasks on Amharic and Tigrinya, with particularly strong gains on out-of-vocabulary entity recognition. Critically, improvements on Amharic and Tigrinya transfer to 17 languages in Africa. VEXMLM demonstrates that vocabulary expansion and tokenizer adaptation provide an effective, computationally efficient path to improve multilingual models for underrepresented languages without retraining from scratch. Resources: GitHub repository | Hugging Face models