Teacher Promotion and Progressive Shrinking: a Framework for Iterative Distillation
Deep neural networks often achieve high accuracy, but their compute, memory, and power demands can hinder deployment on resource-constrained edge devices. This paper proposes an iterative, constraint-aware knowledge distillation framework that compresses a large model into a small one under an explicit model-size budget and a maximum allowable accuracy drop. Starting from user-specified constraints, the method trains candidate students, checks feasibility, and adjusts capacity across generations; oversized students may be promoted to intermediate teachers for subsequent distillation stages. Experiments on MNIST and Speech Commands validate the approach: on MNIST, a 100 KB target with a 2-point accuracy-drop budget yields a 55 KB student achieving 97.28% accuracy; on Speech Commands, a 4-point drop yields a 197.5 KB student reaching 94.08% accuracy. These results demonstrate predictable tradeoffs between compactness and accuracy, and show that multistage teacher promotion can bridge large capacity gaps while preserving performance within user-defined tolerances.