Can R\'enyi Accountants Beat Lossless Accountants for DP-SGD?
Abstract
Recent frameworks in differential privacy (DP), such as privacy loss distributions and $f$-DP, provide lossless representations of the privacy leakage of mechanisms. These approaches are often considered to yield tighter $(\varepsilon, \delta)$-DP guarantees than those obtained from relaxed notions such as R\'enyi DP. But is that always the case? We study this question for differentially private stochastic gradient descent (DP-SGD) with fixed-size batches under replace adjacency. Our key observation is that in this setting, privacy accounting based on R\'enyi DP can use a pair of worst-case distributions (i.e., a dominating pair) that differs from the one used in the lossless frameworks. In particular, we characterize the worst-case distributions as two-dimensional Gaussian mixtures and derive an improved R\'enyi DP bound for DP-SGD. Numerical and asymptotic comparisons show that across various hyperparameter settings, a R\'enyi DP accountant based on the new bound can yield better privacy guarantees than those obtained from the lossless frameworks.
