Continually Post-training Small Models via Synthetic Self-distillation
Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-follo...
Woosung Koh, Hyunsoo Lee, Kyungjae Lee et al.
· 0 citations