TL;DR
RightFeatKD is proposed, a novel feature-based distillation framework that captures rich semantic knowledge from the intermediate representations of the teacher model using a specialized loss function, ensuring reliable and meaningful supervision.
Knowledge distillation (KD) has become a pivotal technique for transferring knowledge from large-scale teacher models to lightweight student models. However, traditional feature-based distillation methods necessitate the direct exposure of the teacher’s intermediate representations, raising concerns regarding data privacy and the leakage of proprietary model details. These concerns often hinder the deployment of distillation in collaborative or cloud-based scenarios. To address these challenges, we propose a privacy-aware teacher-oriented projected feature distillation framework that reduces direct feature exposure while preserving distillation effectiveness. Specifically, our method employs a low-rank projection strategy to obfuscate the teacher’s features into a compact subspace. Authorized student models leverage this projection matrix to align their own features, enabling effective knowledge transfer while reducing direct exposure of the teacher’s original spatial patterns. Extensive experiments on benchmark datasets demonstrate empirical feature obfuscation and reconstruction resistance while maintaining competitive performance in object detection and semantic segmentation, with gains in several settings even when learning from projected representations.
This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks, showing that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
This work revisits on-policy Reverse Kullback-Leibler distillation and decomposes its objective into a teacher-fitting term and a student-entropy term, without introducing an explicit FKL branch, and proposes Adaptive Entropy Distillation (AED), which uses the teacher's entropy to dynamically calibrate token-level imitation strength.
Knowledge distillation (KD) from cross-encoder teachers is a widely adopted technique for training effective neural retrieval models. However, recent studies have revealed a puzzling phenomenon: while retrieval models trained with contrastive loss (CL) exhibit clear scaling behavior with larger language models, KD-trained models show minimal performance gains as model size increases from 1B to 8B parameters. The underlying cause of this scaling failure remains unexplored. In this work, we hypothesize that the teacher model's capacity acts as an information bottleneck, limiting how much large student models can learn. To test this hypothesis, we conduct systematic experiments using decoder-only LLMs (Llama-3: 1B, 3B, 8B) as student retrievers and cross-encoder teachers ranging from 66M to 3B parameters. Our experiments on MSMARCO and BEIR benchmarks reveal that: (1) Small teachers severely constrain student scaling, with 1B, 3B, and 8B students performing nearly identically. (2) Larger teachers progressively restore scaling behavior, enabling significant performance gains at the 8B scale. (3) A teacher-to-student parameter ratio above a critical threshold appears necessary for effective knowledge transfer. Our findings provide practical guidance for selecting appropriate teacher models when training large-scale neural retrievers with knowledge distillation.