DisagreementNet: Lightweight Multi-Teacher Distillation for Retinal OCT Classification—A Leakage-Aware Evaluation

Abstract

Retinal disease classification from optical coherence tomography usually relies on large pretrained backbones that do not fit the hardware available in most clinics. This paper presents DisagreementNet, a 227,557-parameter Convolutional Neural Network (CNN) trained from scratch, and Disagreement-Aware Knowledge Distillation (DA-KD), a loss that transfers knowledge from an EfficientNetB3 CNN teacher and a DeiT-Small Vision Transformer teacher by weighting each sample by the Jensen-Shannon divergence between their outputs. The OCTDL dataset’s 2,064 images come from 820 patients, so an image-level split leaves 79.3% of test images sharing a patient with the training set. All results here use patient-grouped five-fold cross-validation, pooled so that every image is scored exactly once by a model that has not seen its patient. Under this protocol the network reaches 0.888 accuracy with hard-label training, above the ResNet50 (0.846) and VGG16 (0.859) accuracies published with the dataset, at 110 and 610 times fewer parameters. The proposed components do not survive the same test. DA-KD reaches 0.763 macro F1, lowest of six configurations and indistinguishable from uniform logit averaging; removing the attention module leaves 140,935 parameters and is significantly more accurate (McNemar $p \lt 0.0001$ ). Both teachers score below the same architecture trained with no teacher at all, which removes the premise distillation depends on. Training all six configurations under both protocols attributes a mean 0.058 macro F1 to patient overlap, distributed unevenly: hard-label training gains 0.003, every distillation variant between 0.052 and 0.104. The trained model occupies 0.9 MB in float32, requires 81.7 M multiply-accumulate operations per image, and completes single-thread CPU inference in 6.4 ms via ONNX Runtime, which we report as a deployment-feasible footprint rather than validated on-device performance.