Beyond Prediction-Set Size: Quantifying Conformal Uncertainty through Credal Geometry
TL;DR
Credal geometry is established as a principled framework for quantifying conformal uncertainty and understanding its limitations, reducing potentially exponential computations to linear time after sorting the conformal p-values.
Abstract
Prediction-set size offers a coarse summary of conformal uncertainty. We develop a richer perspective through the geometry of credal sets: families of probability distributions compatible with conformal prediction (CP) regions. We establish this representation for split CP, extending a correspondence recently identified for full CP. Building on this representation, we derive analytical characterisations and exact algorithms for several widely used credal uncertainty measures, reducing potentially exponential computations to linear time after sorting the conformal p-values. We further unify several of these measures with classical conformal summaries by integrating set-size penalties based on Tsallis entropies across error levels. We also prove an impossibility theorem: for common split-conformal regression constructions, uncertainty summaries of the induced credal sets cannot distinguish between inputs. Classification experiments on selective prediction and distribution-shift detection show that these summaries often recover useful ranking information lost by prediction-set size at a fixed error level, enabling finer-grained uncertainty assessment. These results establish credal geometry as a principled framework for quantifying conformal uncertainty and understanding its limitations.
