#artificial intelligence #machine learning #natural language processing Preprint

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Atsuki Yamaguchi Tatsuro Inaba Joel Niklaus Michal Štefánik Aline Villavicencio Nikolaos Aletras

Sep 2026 · 0 citations · 68 references

Computer Science

TL;DR

A comprehensive study on PPT spanning five PPT tasks, four PT data mixtures, four parameter scales (500M to 7B), and PT budgets of up to 100B tokens finds that downstream gains arise from PPT tasks that improve long-range retrieval.

Abstract

Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical prior, i.e., a structural inductive bias learned during PPT that transfers to natural language grammar. However, PPT has only been tested on models of at most 1B parameters and PT budgets below 2B tokens on predominantly web text. It is unknown whether PPT is effective at larger scales and under more realistic PT data mixtures that combine diverse sources (e.g., code and math). We therefore present a comprehensive study on PPT spanning five PPT tasks, four PT data mixtures, four parameter scales (500M to 7B), and PT budgets of up to 100B tokens. Our results demonstrate that the downstream performance and token efficiency gains of PPT persist at scale, e.g., saving at least 21B PT tokens at the 3B scale. However, in contrast to prior work, we find no consistent evidence that these gains stem from a grammatical prior. Downstream performance does not consistently align with grammatical acceptability across model sizes. Instead, we find that downstream gains arise from PPT tasks that improve long-range retrieval. Finally, PPT performance gains are robust to how PT data mixtures are composed and diminish only when web text is absent. Overall, PPT is a low-cost addition to PT, and future PPT task design should target long-range retrieval rather than natural language grammar.

DOI arXiv Semantic Scholar

View source

Similar papers

#machine learning Preprint Sep 2026

Lasting Effects of Abstract Pretraining Beyond Perplexity

The results show that early abstract training can reliably shape the capabilities language models later acquire, and in small language models, such a warm-up improves specific capabilities that are not reflected in language-modeling perplexity.

Zachary Shinnick, Hemanth Saratchandran, Damien Teney et al. · 0 citations

#machine learning Preprint Oct 2026

Conditional Transfer from Controlled Pretraining Mixtures to Code

Synthetic tasks are increasingly used both as probes of language-model capability and as pretraining data. Both uses are often justified by loss reduction: falling loss is treated as informative, and faster loss reduction with more sampling as evidence that a task is worth sampling. We separate three signals. A task is...

Ohad Rubin · 0 citations

#machine learning Preprint Oct 2026

Learning to Learn a Language

We present the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained only on samples from a synthetic non-linguistic prior. Given a prefix of real text, it learns to predict the language in context with frozen weights, having never seen a word of any real language. Every training sequen...

Lennart Carstens-Behrens, Holger Fröhlich · 0 citations

#artificial intelligence Preprint Oct 2026

A Scaled-Up Empirical Study of Syntactic Language Models

Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalization and downstream tasks. Prior studies, however, have mostly evaluated smaller models trained on less pretraining data and tested on narrower task suites. In addition, the...

Peng-Cheng Wang, Zhe-Xu Zhang, Chao Lou et al. · 0 citations

#natural language process... Preprint Oct 2026

More Than Words: Compositional Tokenization for Efficient Language Models

Language models process and generate text sequentially in token units, and the tokenizer determines how much text each inference step covers. Under standard tokenization, a short English phrase such as"On the table."is usually produced as four separate predictions for the preposition (On), article (the), noun (table),...

Yuval Reif, Guy Kaplan, Roy Schwartz · 0 citations

Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs

The experimental results show that even 130M-parameter models benefit from including the MTP task in the pre-training objective, and hold even under severe data constraints, as demonstrated on both zero-shot benchmarks and downstream tasks.

Ansar Aynetdinov, Alan Akbik · 1 citation