Dual distillation and distribution-aware memory replay with self-learning for class-incremental learning

Abstract

Class-Incremental Learning (CIL) aims to enable models to continuously learn new classes while retaining knowledge of previously learned classes. However, due to the severe imbalance between the number of new and old class samples during incremental stages, and the inconsistent supervision mechanisms employed by existing distillation methods for new and old classes, models are prone to developing a significant bias toward new classes, further exacerbating catastrophic forgetting. Moreover, the widely used Herding sample selection strategy primarily relies on class mean information, failing to fully capture the intrinsic diversity structure within classes, and is susceptible to feature drift during continual learning. To address these issues, this paper proposes a dual knowledge distillation with clustering-based memory selection for class-incremental learning. First, a unified knowledge distillation framework for both new and old classes is designed. During the new class learning phase, the ground-truth labels are fused with the teacher model's outputs to construct soft targets that incorporate historical knowledge, alleviating excessive bias toward current classes. During the old class replay phase, a masking mechanism suppresses the teacher model's responses to new classes, guiding the model to focus more on preserving old knowledge. Second, a clustering-based representative sample selection strategy is proposed, which excavates the multi-modal structure within each class to improve the coverage of the replayed samples over the true data distribution and enhance the robustness of the memory buffer to feature drift.Experimental results on the CIFAR-100 dataset demonstrate that the proposed method outperforms several mainstream class-incremental learning methods. Ablation studies further validate the effectiveness of the dual knowledge distillation mechanism and the clustering-based sample selection strategy.