Skip to content
Open access

Sketching-Based Communication-Efficient Federated Low-Rank Adaptation With Differential Privacy

2026 · IEEE Transactions on Networking · Vol 34, pp. 7935-7948 · 0 citations · 45 references

Abstract

With the rapid advancement of edge computing, federated learning (FL) has become a widely adopted paradigm for artificial intelligence (AI) applications, especially when data privacy or limited computational resources prevent centralized training. However, even parameter-efficient methods like Low-Rank Adaptation (LoRA) incur significant communication overhead in FL. Existing approaches to reduce this overhead, such as selectively freezing low-rank matrices, often compromise model expressiveness. To address these challenges, we propose CP-FedLoRA, a cross-layer co-designed framework spanning the model and physical layers for communication-efficient and privacy-preserving federated LoRA. CP-FedLoRA adaptively selects one of the two low-rank matrices for local training and integrates Count Sketch-based compression with AirComp aggregation to further reduce communication overhead and enhance privacy. We theoretically show that the sketching error, together with wireless channel noise, yields a closed-form privacy guarantee under zero-concentrated differential privacy (zcDP). We further establish a non-convex convergence bound for CP-FedLoRA and derive an optimal sketch-configuration strategy to balance privacy and model utility. Extensive experiments validate the effectiveness of CP-FedLoRA on both RoBERTa-base and Llama-3-8B. The results show that CP-FedLoRA consistently reduces communication overhead by 12.6%-19.9% on RoBERTa-base, while achieving comparable accuracy with over 20% reduction in communication overhead on Llama-3-8B, demonstrating its robustness across diverse model architectures and tasks.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.