This work trains small language models from scratch on representative subsets of pretraining data and introduces variants of transformers in which standard activation functions are replaced with learned splines tuned on specific datasets, providing a simple way to modify their inductive biases.
Damien Teney, Liang-Ze Jiang, Zachary Shinnick et al.· 0 citations
This work constructs TypewriterCorpus, a 54B-token historical corpus with extensive temporal filtering, proposes lexically grounded instruction tuning that constrains all responses to vocabulary from historical source documents, and introduces History-Event, a benchmark of 2,344 events for evaluating both competence an...
Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber et al.· 4 citations
The results show that early abstract training can reliably shape the capabilities language models later acquire, and in small language models, such a warm-up improves specific capabilities that are not reflected in language-modeling perplexity.
It is found that procedural pretraining can improve molecular property prediction even after subsequent molecular pretraining, and procedural data can provide transferable structure for molecular learning and offer a complementary route to improving performance when labeled molecular data are limited.
M. Friedemann, Zachary Shinnick, P. Torr et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.