Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining
The results show that learnable subword tokenisation can improve sample-efficiency for BabyLM pretraining and analyse the subword learning dynamics of the models and find that tokenisation gradually converges on subword units that balance morphological alignment and fine-grained segmentation.
Francois Meyer
· 0 citations