Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack
It is shown that this assumption can fail in a full 30B mixture-of-experts training pipeline, and the checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.
Sohir Maskey, Philipp Scholl, Jonas Knupp et al.
· 0 citations