Not All Forgetting Is Equal: Retention Dynamics in Fine-Tuned Image Classifiers

Abstract

Fine-tuning a pretrained classifier leaves some samples reliably learned and others cycling between correct and incorrect. Curriculum learning, data pruning and dataset cartography assume that pattern is a property of the sample, untested. We record per-sample correctness at every epoch while fine-tuning ResNet-18 and DeiT-Small on an imbalanced retinal OCT dataset and CUB-200-2011, matching samples by image identity and holding the split fixed across seeds. Per-sample retention is reproducible: cross-run Spearman correlation of the fitted decay constant is 0.37 to 0.59 over ten seeds. It is architecture-specific: two runs of one backbone agree more than two backbones on identical data (0.45 and 0.59 within against 0.30 between on OCTDL). Loss after five frozen-backbone epochs predicts a different run's decay constant at 0.29 to 0.43. The Ebbinghaus exponential does not survive: monotone decay, the one shape it can represent, is 0.1% to 0.8% of samples, and on traces that do forget mean R-squared is negative in all four configurations. A power law and a free-asymptote variant fail on the same traces: the defect is monotonicity. Across five sampling arms with matched exposure, prioritisation ratios of 2.7x to 28x, and an online variant, none of 48 comparisons against uniform sampling survives Benjamini-Hochberg correction, though three seeds detect only about four accuracy points. A stable, cheap difficulty score does not buy generalisation through sampling. Patient-grouped splitting, the remedy for a leak reaching 76% to 78% of OCT test images, moves that dataset's headline metrics by less than their run-to-run spread.