Skip to content

Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

Sep 2026 · arXiv.org · Vol abs/2609.23916 · 0 citations · 42 references
Computer Science

TL;DR

This work evaluates continued pretraining on FineWeb-Edu dumps drawn strictly from after each model's knowledge cutoff, evaluated across six open-weight models spanning three families and four parameter scales, painting a more optimistic picture of time-incremental CPT than the prior continual learning literature suggests.

Abstract

Large language models (LLMs) drift out of date the moment their pretraining ends, yet retraining from scratch is prohibitively expensive. Continued pretraining (CPT) is the natural remedy, but it is typically evaluated through a continual learning lens that assumes disjoint data streams. This is a poor fit for time-incremental updates on web-scale crawls, where successive snapshots share substantial URL overlap by design. We study time-incremental CPT in this realistic regime: continued pretraining on FineWeb-Edu dumps drawn strictly from after each model's knowledge cutoff, evaluated across six open-weight models spanning three families (OLMo2, Llama-3.1/3.2, Gemma-3-1B) and four parameter scales (1B-3B-7B-8B). We organize our findings around four practical questions. (i) Is knowledge acquired? Yes, but heterogeneously, and without catastrophic forgetting: five of six models also improve on pre-cutoff factual recall, and the gains track pretraining saturation (driven primarily by token budget per parameter). (ii) What does it cost? Almost nothing: the macro-average across a thirteen-task suite stays within 0.01 of the base for every model. (iii) What is the recipe? Data quality dominates quantity (a curated 6B-token slice matches a broader 40B one); the optima for knowledge acquisition and general capability are separated by roughly an order of magnitude in learning rate; and LoRA at sufficient rank matches full CPT. (iv) Does it survive deployment? CPT gains transfer through SFT, while DPO's effect is family-dependent. Together, these results paint a more optimistic picture of time-incremental CPT than the prior continual learning literature suggests.

View source

Similar papers

#machine learning Preprint Oct 2026

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

A Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces, which is substantially more token-efficient than forgetting-aw...

Zheng-Hao Zhao, Gao-Wen Liu, Zhi-Ling Lan et al. · 0 citations
#machine learning Preprint Oct 2026

Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning

Modeling LLM reasoning as search over a directed acyclic graph, a unified theoretical analysis of both the dynamics of post-training---OPSD and supervised fine-tuning (SFT) in continual learning---and the impact of pre-training on subsequent performance shows that reliable continual reasoning depends on how post-traini...

Shinichi Uemura, Taiji Suzuki · 0 citations
#artificial intelligence Review Aug 2026

When to Review: Spaced Repetition for Continual Pre-Training of Language Models

Spaced Repetition Training (SRT) is introduced, a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm, and preserves broad benchmark performance that naive continual pre-training and uniform replay substantially degrade.

Alankar Atreya, Devesh Batra, Yoages Kumar Mantri et al. · 1 citation
Preprint Aug 2026

What to Forget in Unlearning? Forget Set Curation for Language Models

This work introduces CleanSlate, a benchmark for verbatim output suppression over songs and books, with model-specific extraction profiles, content-grounded QA, and capability-retention evaluations, and shows that practical unlearning is not only an optimization problem once a forget set is given, but also what can be...

Animesh Jha, Arpandeep Khatua, Youssef Allouah et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.