Fed-DiffLoRA: Personalized Federated Style Transfer for T2I Diffusion Models

Abstract

Low-Rank Adaptation (LoRA) merging enables efficient customization of T2I diffusion models; nevertheless, centralized aggregation raises serious privacy concerns. While federated LoRA adaptations mitigate these risks, diffusion models present unique challenges: structural heterogeneity and vulnerability to member inference attacks. To overcome these limitations, we propose Fed-DiffLoRA, a privacy-preserving framework that securely aggregates LoRA adapters. At the client level, we disentangle user-specific adaptations into two orthogonal subspaces: content LoRAs preserving semantic fidelity and style LoRAs encoding stylistic features, isolating sensitive attributes from stylistic components. We design a learnable aggregation operator that dynamically optimizes cross-client style LoRA fusion based on the semantic vectors of clients’ content LoRAs, achieving high-fidelity style blending while suppressing client-identifiable patterns. We also provide theoretical guarantees for convergence and privacy. Extensive experiments validate the effectiveness of the proposed approach, achieving substantial reductions in attack success rates and consistently high stylization fidelity.