Analysis of the Efficiency of Subword Tokenizers in a Low-Resource Linguistic Environment: Implementation Experience for the Tajik Language
The experimental results revealed the strengths and weaknesses of various approaches to subword segmentation and identified the most effective tokenization strategies under the conditions of the morphological complexity of the Tajik language.