Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
A comprehensive study on PPT spanning five PPT tasks, four PT data mixtures, four parameter scales (500M to 7B), and PT budgets of up to 100B tokens finds that downstream gains arise from PPT tasks that improve long-range retrieval.
Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus et al.
· 0 citations