Skip to content
Book Open access

OP-CL: On-Policy Guided Continual Post-training for Multimodal Large Language Models

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 4 references

Abstract

Continual post-training is essential for adapting Multimodal Large Language Models (MLLMs) to evolving tasks, but it often leads to catastrophic forgetting. The core challenge is to achieve rapid adaptation to new task formats and semantics while preventing policy drift from accumulating into forgetting during sequential training. Existing paradigms address this trade-off only partially: Supervised Fine-Tuning (SFT) enables rapid adaptation but often disrupts prior knowledge, whereas Reinforcement Learning (RL) better preserves existing capabilities through on-policy updates but typically converges slowly and struggles on open-ended tasks. We propose OP-CL, a two-stage on-policy guided framework for continual post-training in MLLMs. In Stage 1, On-Policy Corrective Supervision transforms model rollouts into corrective supervision, providing efficient learning signals for acquiring new-task behaviors, especially on open-ended tasks. Since such rapid adaptation may still introduce policy deviation, Stage 2, Reward-KL Stabilized Reinforcement further stabilizes continual adaptation by adaptively regulating KL regularization to limit excessive policy updates. The two stages therefore address two complementary requirements of continual post-training: efficient acquisition and stable retention. Experiments show that OP-CL consistently achieves a better balance between new task adaptation and old task retention than representative SFT- and RL-based baselines.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.