Depletion or Contamination? Decomposing Minority Label Noise in the Evaluation of Imbalance-Handling Strategies

Abstract

Class imbalance and unreliable data are usually studied apart, and when they are studied together, minority label noise is treated as a single mechanism. It is not. Recording a genuine positive as a negative both depletes the positive class and contaminates the negative class with an example that contradicts the positives that remain, and a study that reports only the resulting performance drop cannot say which mechanism a method is failing to survive. We separate them with paired corruption arms that share a random draw of affected rows, relabelling the selected positives in one arm and deleting exactly the same rows in the other, so that the gap between the arms isolates the cost of the contradictory labels. Crossing six imbalancehandling strategies with three classifiers over 20 benchmark datasets and one extremely imbalanced fraud dataset, and summarising robustness by the normalised area under the corruption curve, we find that mislabelling a training positive costs 0.038 of clean PR-AUC more than deleting the same row, so a substantial part of the damage is the contradictory label rather than the lost example. Comparing methods with the Friedman test blocked on datasets — rather than on cross-validation folds, a common error that inflates the statistic in proportion to the number of folds — the imbalance strategy separates where the classifier family does not, and only 1 of five strategies separates from applying none at all. Resampling is not free: the most aggressive strategy degrades the Brier score to 4.5× that of the unmodified baseline, a cost invisible to threshold-free discrimination metrics. The decomposition costs one experimental arm, applies to any study that injects minority label noise, and turns a measurement of how much a method degrades into a statement about what it is failing to survive