Zi-Yue Wang

Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

The repetition count that works best for a small language model may not remain best at a larger scale. We study this effect in pretraining with a finite target corpus mixed with generic data at a fixed target fraction. On Wikipedia-derived data and Proof-Pile-2, the ranking of measured repetition counts changes with mo...