#machine learning
Sep 2026
Lasting Effects of Abstract Pretraining Beyond Perplexity
The results show that early abstract training can reliably shape the capabilities language models later acquire, and in small language models, such a warm-up improves specific capabilities that are not reflected in language-modeling perplexity.
Zachary Shinnick, Hemanth Saratchandran, Damien Teney et al.
· arXiv.org · 1 citation