PTFTL: Personalized Transformer-Based Federated Transfer Learning With Projection Layer for Malware Detection in Edge--IoT Networks

Abstract

The increasing deployment of Edge–Internet of Things (IoT) networks intensifies the need for robust and privacy-preserving malware detection solutions. However, such environments are characterized by highly heterogeneous and nonindependent and identically distributed (non-IID) data distributions, limited computational resources, and strict privacy constraints, which collectively hinder the effectiveness of conventional centralized and federated learning (FL) approaches. Although transformer models and FL have independently shown promise for intrusion detection, their joint potential under statistical heterogeneity remains insufficiently explored. This work proposes a personalized transformer-based federated transfer learning (PTFTL) framework designed for efficient and adaptable malware detection in Edge–IoT environments. The proposed model introduces a lightweight projection layer that enables effective attention over raw traffic data while reducing computational burden. Personalization is achieved by keeping selected model components local to each device, thereby improving robustness under heterogeneous data distributions. Experimental results demonstrate that PTFTL substantially improves stability and detection performance under severe non-IID conditions. In particular, the inclusion of the projection layer enhances resilience to model collapse by up to 38%, whereas replacing it with a traditional embedding reduces stability by at least 10%, highlighting the critical role of the projection design in federated transformer models.