Knowledge Distillation Across Tasks and Model Families: A Comparative Simulation Study—Denoising, Dark Knowledge, and the Role of Model Capacity

Abstract

Knowledge distillation transfers information from a high-capacity teacher to a smaller student, but its behavior across regression, classification, and heterogeneous tabular model families remains insufficiently understood. This paper presents a comparative simulation study of distillation in structured-data settings, where neural networks, tree ensembles, kernel methods, nearest-neighbor methods, and linear models are plausible competitors. Synthetic datasets with known ground truth and controlled noise are used to evaluate a three-model teacher ensemble distilled into multilayer perceptrons, tree ensembles, decision trees, kernel methods, instance-based methods, and linear models. Distillation weights and temperatures are selected on validation data, and test results are reported with confidence intervals, paired tests, and multiplicity-control checks. In regression, distillation acts as target smoothing and denoising: the teacher reduces label noise by 17.4%, 13 of 14 students improve numerically, and the MSE reduction is 7.1%. After multiplicity control, robust gains concentrate among MLPs. In classification, gains are smaller: 12 of 15 students improve, with an average accuracy gain of 0.5 percentage points, and soft-label distillation improves calibration for MLPs. Additional noise sweeps, teacher ablations, real-data checks, baseline comparisons, temperature analysis, and calibration results show that distillation helps most when the student has sufficient capacity, and the teacher provides a cleaner or more informative target. An extended set of confirmatory analyses—including teacher-quality controls with a shuffled negative control, ensemble-based uncertainty proxies, harder data conditions, distribution-shift robustness, a capacity ladder, computational-cost analysis, additional baselines and calibration metrics, an α–T factorial analysis, quantitative measures of soft-probability structure, and comparisons with modern tabular models—shows that useful transfer requires teacher predictions that remain conditionally aligned with the inputs. The classification gains are small in absolute accuracy but stable across seeds and are accompanied by improvements in selected calibration measures.