FedMuse: A Privacy-Aware Federated Framework for Multi-Style Text-to-Image Art Generation
TL;DR
FedMuse is proposed, a privacy-aware federated framework for multi-style text-to-image art generation, in which distributed clients collaboratively train a shared generative model while keeping their private art data local, and an adaptive privacy regularizer is introduced to reduce memorization risk while maintaining visual quality and prompt consistency.
Abstract
Text-to-image generation has rapidly advanced the creation of digital artwork, yet most existing models rely on centralized training pipelines that require collecting large-scale image–text pairs from artists, design studios, or online communities. Such centralized practice raises serious privacy, ownership, and style leakage concerns, especially when local datasets contain identifiable artistic signatures or proprietary visual assets. To address this problem, this paper proposes FedMuse, a privacy-aware federated framework for multi-style text-to-image art generation, in which distributed clients collaboratively train a shared generative model while keeping their private art data local. The proposed framework decomposes the learning process into three coordinated components: a global semantic alignment module that captures cross-client text–image correspondence, a local style adapter that preserves client-specific artistic characteristics, and a privacy-calibrated aggregation mechanism that suppresses sensitive style leakage during model update exchange. To further improve multi-style generation, we design a style-disentangled federated optimization algorithm that separates content-relevant knowledge from client-private stylistic representations, allowing the global model to generalize across diverse artistic domains without directly absorbing private local styles. In addition, an adaptive privacy regularizer is introduced to reduce memorization risk while maintaining visual quality and prompt consistency. Experiments on real-world text-to-image art datasets demonstrate competitive generation quality, stronger personalization, and improved empirical resistance to membership and style-leakage attacks. Because the calibration statistics are data-dependent, FedMuse does not claim a certified end-to-end differential-privacy budget. The results suggest that privacy-aware collaboration can be a practical direction for distributed AI art generation.
