Sketching-Based Communication-Efficient Federated Low-Rank Adaptation With Differential Privacy
Abstract
With the rapid advancement of edge computing, federated learning (FL) has become a widely adopted paradigm for artificial intelligence (AI) applications, especially when data privacy or limited computational resources prevent centralized training. However, even parameter-efficient methods like Low-Rank Adaptation (LoRA) incur significant communication overhead in FL. Existing approaches to reduce this overhead, such as selectively freezing low-rank matrices, often compromise model expressiveness. To address these challenges, we propose CP-FedLoRA, a cross-layer co-designed framework spanning the model and physical layers for communication-efficient and privacy-preserving federated LoRA. CP-FedLoRA adaptively selects one of the two low-rank matrices for local training and integrates Count Sketch-based compression with AirComp aggregation to further reduce communication overhead and enhance privacy. We theoretically show that the sketching error, together with wireless channel noise, yields a closed-form privacy guarantee under zero-concentrated differential privacy (zcDP). We further establish a non-convex convergence bound for CP-FedLoRA and derive an optimal sketch-configuration strategy to balance privacy and model utility. Extensive experiments validate the effectiveness of CP-FedLoRA on both RoBERTa-base and Llama-3-8B. The results show that CP-FedLoRA consistently reduces communication overhead by 12.6%-19.9% on RoBERTa-base, while achieving comparable accuracy with over 20% reduction in communication overhead on Llama-3-8B, demonstrating its robustness across diverse model architectures and tasks.