MoirfEolas and Cr\'iochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology
It is found that the Unigram Language Model aligns with Irish morphology more often than the other algorithms evaluated, and trade-offs between morphological-alignment of tokenization with both compression as well as vocabulary efficiency are found, providing practical insights for Irish natural language processing dev...