CoLT-FL: Compressed Lightweight Transformer-based Federated Learning for Edge Intelligence

Abstract

The edge devices generate a tremendous amount of sensitive data, which makes the centralized model of training difficult to implement. In this regard, federated learning is introduced, which can perform the task of model training across multiple devices without the need for sharing data, although communication overhead is introduced. The Transformer model is known for its superior learning ability, although the computational cost makes it less applicable for edge devices. Therefore, the need for the proposed CoLT-FL, which is a federated learning framework using a compressed lightweight Transformer model, is introduced. The sparsity-based attention mechanism is introduced, which not only minimizes communication overhead but retains the relevant data as well. The observations made during the experiment indicate that the proposed model performs faster, minimizes latency, and increases the overall accuracy.