Near-Optimal Machine Unlearning Utility for Smooth Strongly Convex Losses
TL;DR
This work nearly resolves the fundamental statistical cost of unlearning by proving upper and lower bounds on the excess population risk of approximate $(\varepsilon, \delta)$-unlearning; the authors' bounds are tight up to a condition-number factor.
Abstract
Machine unlearning is motivated by legal and user-facing requirements to remove the influence of individuals'data from trained models, such as the right to be forgotten. Prior work has developed algorithms and error bounds for unlearning in smooth strongly convex stochastic optimization but the fundamental statistical cost of unlearning has remained unclear. We nearly resolve this problem by proving upper and lower bounds on the excess population risk of approximate $(\varepsilon, \delta)$-unlearning; our bounds are tight up to a condition-number factor. For mean estimation over the unit ball, our upper and lower bounds match. In fact, our algorithm achieves $\varepsilon$-unlearning, which implies a notable separation between differential privacy and unlearning: $(\varepsilon, \delta)$-unlearning has no statistical advantage over pure $\varepsilon$-unlearning. The optimal rate is the usual sampling error plus an unlearning penalty that interpolates between the retraining from scratch rate and an exponentially smaller term as $\varepsilon/d$ grows, where $d$ is the dimension of the model. The retraining penalty dominates the sampling error for large unlearning requests. In particular, retraining from scratch is information theoretically optimal up to $\varepsilon \lesssim d$. On the other hand, for $\varepsilon \gg d$ and large unlearning requests, our $\varepsilon$-unlearning algorithm offers an exponential accuracy improvement over retraining the model from scratch and differentially private baselines.
