Efficient Privacy Auditing for Generative Model via Local Information

Abstract

Diffusion models have become the dominant approach for text-to-image generation, but their ability to memorize training data raises increasing concerns about privacy leakage. Differential privacy (DP) is widely adopted to mitigate such privacy risks during model fine-tuning, yet the practical privacy leakage of differentially private diffusion models remains difficult to assess, especially in black-box settings where only generated images are observable. Existing auditing methods for diffusion models typically rely on membership inference attacks based on whole-image similarity between generated samples and target images. However, such approaches may underestimate privacy leakage when similarity between generated images and training samples is concentrated in localized image regions rather than at the whole-image level. In this work, we explore privacy leakage in diffusion models fine-tuned with differential privacy from a black-box perspective. We propose an empirical privacy assessment framework that leverages local image information, instead of treating images as indivisible wholes, to improve the distinguishability of privacy leakage signals. To improve privacy auditing efficiency and reduce sampling variance, we further leverage the image inpainting interface of diffusion models to perform region-focused auditing in a fully black-box setting. Extensive experiments across different fine-tuning and auditing settings demonstrate that our approach provides more reliable empirical assessments of privacy leakage than whole-image-based auditing methods.