A Knowledge-Imparting Generative Modelling Framework for Heterogeneous Federated Learning

Abstract

Federated learning aims to provide security for client data privacy in practical machine learning applications. In principle, a global server aggregates the models produced by local clients to obtain a global model. However, the server is challenged when collaborating with local clients handling non-identically distributed data without authorisation to access it. Therefore, advanced solutions advocate the use of generative modules to deliver surrogate data to local clients during a server-agent interaction, without revealing private particulars. We argue that such a unidirectional transfer of surrogate patterns cannot fully represent and harmonise knowledge during the server-client interactions. To this end, we propose a knowledge-imparting generative modelling framework (FedKIG) based on adversarial feature learning and bidirectional knowledge distillation, to explore the potential of interactive generative modelling. In particular, Fed-KIG trains a feature discriminator for each local client to identify the surrogate patterns extracted by the global model. Under the supervision of the local feature discriminators, the server learns a global generator to generate pseudo samples that convey its global perspective. In this manner, local models are enabled to absorb global knowledge, thereby mitigating the training data divergence caused by data heterogeneity. In addition, we develop a bidirectional knowledge distillation strategy to support the entire learning process. This strategy breaks the rigidity of federated distillation by updating knowledge transfer between the server and the clients iteratively, thus overcoming the learning-forgetting issue. The proposed privacy-protected server-client interaction solution supports explicit knowledge generation for exploitation in federated learning. Extensive experimental results indicate that FedKIG significantly improves the generalisation performance and the stability of the model in heterogeneous federated learning scenarios.