Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives
TL;DR
It is suggested that simple in-training interpolation deserves consideration before introducing more elaborate forgetting-mitigation mechanisms, and Sequential Fine-tuning with Averaging with Averaging with SFA is introduced and evaluated.
Abstract
Have we made mitigating catastrophic forgetting more complicated than it needs to be? Continual adaptation often degrades performance on earlier tasks, motivating rehearsal buffers, auxiliary objectives, and adaptive merging procedures. We investigate how far a simpler alternative-periodic parameter interpolation during fine-tuning-can go on its own. We introduce Sequential Fine-tuning with Averaging (SFA) and evaluate it alongside replay-free Lookahead, originally designed for optimization stability to find compelling results. In Qwen3-14B adapter continual learning, fixed configurations of both methods outperform the tested replay and adaptive-merging configurations in mean final accuracy using up to 36.3% less task training time. Furthermore, across language and vision tasks, our results compete with more complex continual-learning methods. Together, these findings suggest that simple in-training interpolation deserves consideration before introducing more elaborate forgetting-mitigation mechanisms.
