A Language Model from 1913: Pretraining on Historical Text
This work constructs TypewriterCorpus, a 54B-token historical corpus with extensive temporal filtering, proposes lexically grounded instruction tuning that constrains all responses to vocabulary from historical source documents, and introduces History-Event, a benchmark of 2,344 events for evaluating both competence an...
Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber et al.
· 4 citations