TL;DR
RightFeatKD is proposed, a novel feature-based distillation framework that captures rich semantic knowledge from the intermediate representations of the teacher model using a specialized loss function, ensuring reliable and meaningful supervision.
Knowledge distillation (KD) has become a pivotal technique for transferring knowledge from large-scale teacher models to lightweight student models. However, traditional feature-based distillation methods necessitate the direct exposure of the teacher’s intermediate representations, raising concerns regarding data priv...
Streamlined Knowledge Distillation is proposed, a simple yet effective logit-based method that transfers only two essential forms of knowledge without requiring additional alignment or relational modeling and introduces a Mahalanobis distance-based direction-wise loss stabilized through Tikhonov regularization and Chol...
This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks, showing that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
This work revisits on-policy Reverse Kullback-Leibler distillation and decomposes its objective into a teacher-fitting term and a student-entropy term, without introducing an explicit FKL branch, and proposes Adaptive Entropy Distillation (AED), which uses the teacher's entropy to dynamically calibrate token-level imit...
This work hypothesizes that the teacher model's capacity acts as an information bottleneck, limiting how much large student models can learn, and provides practical guidance for selecting appropriate teacher models when training large-scale neural retrievers with knowledge distillation.
As a paradigm in continual learning, class incremental learning (CIL) aims to assimilate tasks with mutually exclusive label spaces in sequence while preserving previously established knowledge. Mitigating forgetting in CIL fundamentally relies on transferring knowledge across tasks. A straightforward exemplar-based ap...
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
