Jul 2026· IEEE International Conference on Circuits and Systems for Communications· pp. 1-7· 0 citations· 18 references
Abstract
Deep neural networks often achieve high accuracy, but their compute, memory, and power demands can hinder deployment on resource-constrained edge devices. This paper proposes an iterative, constraint-aware knowledge distillation framework that compresses a large model into a small one under an explicit model-size budget and a maximum allowable accuracy drop. Starting from user-specified constraints, the method trains candidate students, checks feasibility, and adjusts capacity across generations; oversized students may be promoted to intermediate teachers for subsequent distillation stages. Experiments on MNIST and Speech Commands validate the approach: on MNIST, a 100 KB target with a 2-point accuracy-drop budget yields a 55 KB student achieving 97.28% accuracy; on Speech Commands, a 4-point drop yields a 197.5 KB student reaching 94.08% accuracy. These results demonstrate predictable tradeoffs between compactness and accuracy, and show that multistage teacher promotion can bridge large capacity gaps while preserving performance within user-defined tolerances.
Diffusion models have demonstrated remarkable performance across a wide range of generative tasks; however, their high sampling cost remains a critical bottleneck. To address this, consistency distillation (CD) was proposed, offering a reduction in sampling cost by distilling a pretrained diffusion model. However, achieving generative quality comparable to diffusion models requires extensive training for the distillation process, posing a substantial computational challenge. In this article, we introduce variance-reduced consistency learning (vrCL), a novel distillation technique that enables stable and efficient training of consistency models without relying on teacher model evaluations. By leveraging a student-guided sample pair, vrCL ensures training stability while significantly reducing computational costs. This design eliminates the need for repeated teacher model evaluations during training, resulting in high computational efficiency and significantly reduced training time. Empirical results demonstrate that vrCL achieves competitive generative performance with high training efficiency, reaching strong results within just 100k training iterations.
Deploying deep learning models on resource-constrained edge devices necessitates a trade-off between computational efficiency and robustness. While Knowledge Distillation (KD) is widely used to compress large models into lightweight students, the impact of the Teacher’s architectural inductive bias on the Student’s safety profile remains underexplored. This paper investigates Cross-Architecture Distillation, specifically examining whether the robustness properties of a Vision Transformer (ViT) Teacher can be transferred to a Convolutional Neural Network (CNN) Student (MobileNetV2). Using Centered Kernel Alignment (CKA) and Fourier Spectral Analysis, we demonstrate that robustness transfer is driven by the learning of global shape biases— preserving the teacher's functional geometry—rather than strict feature mimicry. To isolate architectural bias from model capacity confounds and validate scalability to high-resolution data, our experiments are conducted on ImageNet-1k. Results reveal that a MobileNetV2 distilled from a ViT-B/16 Teacher achieves a significantly lower mean corruption error (mCE) compared to one distilled from a massive, capacity-matched ResNet-152 (p < 0.0001, Cohen's d = 21.50), despite comparable clean accuracy. Furthermore, we show that the ViT-distilled student effectively suppresses high-frequency noise, validating its resilience against adversarial perturbations. These findings establish a pareto-optimal strategy for deploying robust, lightweight models, proving that the choice of Teacher architecture is a critical hyperparameter for safety-critical edge applications.
Tamuno Opubo Dappa, Somtochukwu Anunobi· International Conference on...· 0 citations
Shift-Augmented Knowledge Distillation (SAKD) is proposed, a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation, enabling single-stage training while producing adaptive, diverse views through a parameter-free cyclic shift.
Xuyi Yu, Yaohua Liu, Chengjun Li et al.· 0 citations
Leading open text-to-image models often carry complementary strengths: one may lead on preference-aligned aesthetics while another follows compositional instructions more faithfully. However, differences in their autoencoders and noise schedules make it difficult to transfer these strengths across models. In this paper, we present Poly-OPD, a framework that can consolidate complementary strengths of heterogeneous teachers into a single compact flow-matching student. To bridge the incompatible latent spaces of different teachers, Poly-OPD performs on-policy distillation through a pixel bridge. Each student-generated image is re-encoded by a selected teacher's encoder and refined from a noise level matched by magnitude under the teacher's noise schedule. The resulting target is further matched to the student in frozen DINOv2 space, enabling supervision across incompatible latent spaces. To retain complementary capabilities without cross-teacher interference, Poly-OPD uses a gradient compatibility diagnostic to organize its adapters: attention LoRA modules are shared across teachers, whereas feed-forward adapters remain teacher-specific. During distillation, a gap-aware curriculum devotes more training to compositional categories where the student still falls short of the teacher. As each gap narrows, training shifts toward categories with larger remaining gaps. By distilling FLUX.1-dev and Z-Image into a 2.5B SD3.5-Medium student, Poly-OPD improves GenEval from 67.3 to 73.3, surpassing both larger teachers, and raises DrawBench HPSv3 from 9.34 to 11.35, consolidating both strengths within a switchable model.
This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks, showing that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner et al.· 0 citations
A practitioner's study of how to make distillation training efficient is presented, organised around two systems contributions, and a fused, chunked KL loss is introduced, making peak memory linear in the sequence length.
Bakbergen Ryskulov, Iker Garc'ia-Ferrero, David Montero et al.· 0 citations