Suit the Remedy to the Case: Precise Backdoor Elimination for Text-to-Image Diffusion Models via Model Editing

Abstract

With the widespread adoption of Text-to-Image (T2I) Diffusion Models (DMs) across various applications, their security has attracted increasing attention. In particular, these models are highly vulnerable to backdoor attacks, where adversaries implant hidden triggers during training to manipulate model outputs at inference time. To deal with the backdoor attacks, existing defense methods typically require substantial training data and employ coarse-grained alignment, leading to catastrophic forgetting and semantic degradation, respectively. To defend against backdoor attacks while preventing performance degradation, this paper first analyzes the behavior of poisoned models and finds that i) the neuron activation variation distributions of trigger tokens are significantly different from those of normal tokens, and ii) the embeddings of non-semantic trigger tokens are far from the manifold space constructed by all the normal tokens in the vocabulary. Based on the findings, we propose ManiEdit, a novel model-edit-based backdoor elimination framework for T2I DMs. Specifically, we propose an activation-based module localization methodology to pinpoint the backdoored module by calculating the Neuron Activation Variations (NAV), enabling precise parameter recoupling within restricted areas to safeguard benign parameters from collateral damage. To sever malicious associations and restore the original semantics of triggers, we present a manifold-driven trigger categorization strategy, which detects triggers via NAV within the localized modules, subsequently applying syntactic and geometric constraints to distinguish non-semantic from semantic triggers. By aligning non-semantic triggers with empty prompts and realigning semantic triggers with denoising-optimized semantic anchors, ManiEdit can achieve efficient elimination without expensive retraining. Comprehensive experiments demonstrate the effectiveness of ManiEdit in defending against various state-of-the-art backdoor attack methods.