Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
This work evaluates continued pretraining on FineWeb-Edu dumps drawn strictly from after each model's knowledge cutoff, evaluated across six open-weight models spanning three families and four parameter scales, painting a more optimistic picture of time-incremental CPT than the prior continual learning literature sugge...