This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks, showing that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
Abstract
The growing size of Convolutional Neural Networks has led to increasingly large and costly models. Knowledge Distillation (KD) addresses this by transferring knowledge from a large network (teacher) to a small one (student), also reducing the training data required. KD is traditionally applied only at the network's final output. However, its behaviour when applied at intermediate network layers has received little attention. This raises the question of whether intermediate block-wise KD, which provides supervision throughout the network, could offer an advantage under specific conditions, such as few instances per class, which is common in fine-grained datasets. This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks. Across eleven datasets, we show that on classic datasets, distilling only the last block is sufficient -- and often best--, whereas fine-grained, data-scarce settings benefit substantially from intermediate supervision, with even a single additional distillation point narrowing the gap considerably. We further study how this supervision should be guided, exploring configurations of varying granularity and informed by an explainability analysis based on attention maps, Centered Kernel Alignment, and Grad-CAM, alongside the impact of teacher and student fine-tuning strategies. This work shows that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
This study introduces a novel memory-augmented self-learning framework that extracts and provides diverse learning sources for adaptive knowledge distillation from the student model itself, resulting in a 2.5-6% increase in accuracy across various benchmark datasets compared to current GNN training and self-distillation methods.
Saurabh Sharma, Souvik Chowdhury, Joydeep Chandra· Data mining and knowledge di...· 0 citations
Knowledge distillation (KD) from cross-encoder teachers is a widely adopted technique for training effective neural retrieval models. However, recent studies have revealed a puzzling phenomenon: while retrieval models trained with contrastive loss (CL) exhibit clear scaling behavior with larger language models, KD-trained models show minimal performance gains as model size increases from 1B to 8B parameters. The underlying cause of this scaling failure remains unexplored. In this work, we hypothesize that the teacher model's capacity acts as an information bottleneck, limiting how much large student models can learn. To test this hypothesis, we conduct systematic experiments using decoder-only LLMs (Llama-3: 1B, 3B, 8B) as student retrievers and cross-encoder teachers ranging from 66M to 3B parameters. Our experiments on MSMARCO and BEIR benchmarks reveal that: (1) Small teachers severely constrain student scaling, with 1B, 3B, and 8B students performing nearly identically. (2) Larger teachers progressively restore scaling behavior, enabling significant performance gains at the 8B scale. (3) A teacher-to-student parameter ratio above a critical threshold appears necessary for effective knowledge transfer. Our findings provide practical guidance for selecting appropriate teacher models when training large-scale neural retrievers with knowledge distillation.
Shuran Zhou, Rui Ling, Junan Chen et al.· Annual International ACM SIG...· 0 citations
Data-free knowledge distillation transfers the knowledge encoded in a teacher model to a student model without access to the original training data. Prior work such as Contrastive Abductive Knowledge Extraction (CAKE) achieves this for classifiers by synthesizing samples near the teacher's decision boundary. In this work, we investigate whether this boundary-seeking principle extends to autoencoder distillation through experiments on the MNIST dataset . To enable a direct comparison, we reformulate continuous reconstruction as a dense, per-feature classification task, allowing the decoder to output categorical logits. We show that boundary-seeking objectives are fundamentally ill-posed in bottlenecked generative architectures. CAKE operates on a single, instance-level objective, but a decoder acts as an array of tightly coupled, feature-level classifiers constrained by a shared low-dimensional bottleneck. Independently sampling contrastive targets for these coupled outputs violates the geometry of the learned latent manifold and produces severe gradient conflicts instead of informative boundary samples. Manifold-aware synthesis bypasses these conflicts entirely and establishes an effective baseline for data-free generative distillation.
A practitioner's study of how to make distillation training efficient is presented, organised around two systems contributions, and a fused, chunked KL loss is introduced, making peak memory linear in the sequence length.
Bakbergen Ryskulov, Iker Garc'ia-Ferrero, David Montero et al.· 0 citations
Deploying deep learning models on resource-constrained edge devices necessitates a trade-off between computational efficiency and robustness. While Knowledge Distillation (KD) is widely used to compress large models into lightweight students, the impact of the Teacher’s architectural inductive bias on the Student’s safety profile remains underexplored. This paper investigates Cross-Architecture Distillation, specifically examining whether the robustness properties of a Vision Transformer (ViT) Teacher can be transferred to a Convolutional Neural Network (CNN) Student (MobileNetV2). Using Centered Kernel Alignment (CKA) and Fourier Spectral Analysis, we demonstrate that robustness transfer is driven by the learning of global shape biases— preserving the teacher's functional geometry—rather than strict feature mimicry. To isolate architectural bias from model capacity confounds and validate scalability to high-resolution data, our experiments are conducted on ImageNet-1k. Results reveal that a MobileNetV2 distilled from a ViT-B/16 Teacher achieves a significantly lower mean corruption error (mCE) compared to one distilled from a massive, capacity-matched ResNet-152 (p < 0.0001, Cohen's d = 21.50), despite comparable clean accuracy. Furthermore, we show that the ViT-distilled student effectively suppresses high-frequency noise, validating its resilience against adversarial perturbations. These findings establish a pareto-optimal strategy for deploying robust, lightweight models, proving that the choice of Teacher architecture is a critical hyperparameter for safety-critical edge applications.
Tamuno Opubo Dappa, Somtochukwu Anunobi· International Conference on...· 0 citations