EDGE: a closed-form directed test for the calibration of probabilistic binary classifiers

Abstract

A probabilistic binary classifier is judged almost everywhere by discrimination - accuracy, the ROC curve, the area under it. Every such criterion is invariant to a monotone distortion of the predicted probabilities, so a classifier can rank perfectly and still return probabilities that are badly wrong. Calibration is the property decisions need, and the field's instrument for it, the binned expected calibration error with its reliability diagram, is descriptive: it has no null distribution, so it cannot say whether the miscalibration it displays is real or noise, and it depends on the binning. We propose EDGE, a calibration test for the canonical probabilistic classifier, logistic regression. EDGE reads the same binned predicted-versus-observed table a reliability diagram plots, and projects its standardized bin residuals onto a small pre-specified basis of smooth calibration-distortion shapes. Its null distribution is a weighted sum of chi-square variables in closed form, costing one pass over the data and one small eigendecomposition: no refit, no resampling, no tuning, so it can run inside cross-validation loops. Binning also makes it robust to the sparsity continuous features create. Across link and feature misspecification the pre-specified default led or tied every rival binned test on the fitted index in 19 of 22 detectable scenarios, and stayed computable where the refit-based Stukel score test separates in 20% to 28% of sparse samples. Its honest limit is rough, high-frequency miscalibration, where omnibus statistics win - a limit an elementary resolution argument shows is shared by every binned instrument, the calibration error included.