Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· 0 citations· 25 references
TL;DR
i-FedLoRA provides privacy guarantees, improves model accuracy by up to 3.8%, and expedites training by 1.37-2.23×, and facilitates heterogeneous LoRA aggregation that selectively prioritizes high-confidence knowledge to filter DP-induced noise, thereby achieving robust knowledge transfer.
Abstract
Federated low-rank adaptation (FedLoRA) allows multiple clients to collaboratively fine-tune large language models (LLMs) on downstream tasks without exposing their private data. To mitigate privacy leakage during aggregation, differential privacy (DP) is widely used to clip and perturb local model updates with noise, yet it can compromise model accuracy due to the inherent privacy-utility trade-off. The performance degradation becomes worse under the FedLoRA setting with the amplified DP noise impact and client heterogeneity in both model structure and data distribution. In this work, we propose iP-FedLoRA, a privacy-preserving federated fine-tuning framework for heterogeneous clients that strikes a good privacy-utility balance. Specifically, to fully utilize clients' heterogeneous resources, we customize LoRA modules based on their available resources. iP-FedLoRA employs matrix-wise differentially private local fine-tuning with sensitivity-aware noise allocation and rank-compensated LoRA regularization, which effectively alleviates noise impact of low-rank modules and enhances training efficiency. By leveraging noise-resilient knowledge distillation, iP-FedLoRA facilitates heterogeneous LoRA aggregation that selectively prioritizes high-confidence knowledge to filter DP-induced noise, thereby achieving robust knowledge transfer. Through rigorous privacy analysis and extensive experiments, we show that iP-FedLoRA provides privacy guarantees, improves model accuracy by up to 3.8%, and expedites training by 1.37-2.23×.
Differential privacy (DP) mechanisms have been widely adopted in federated learning (FL) to enhance model security. However, existing approaches predominantly employ uniform privacy budgets, neglecting personalized requirements arising from heterogeneous user privacy preferences. Such uniform privacy configurations typically necessitate compliance with the most stringent budget, which not only leads to the wasteful underutilization of privacy budgets for certain clients but also compromises overall model utility. To address this limitation, we propose FedSPA, a Subspace Projection Aggregation personalized differential private Federated learning framework. The proposed method conducts singular value decomposition operations on noise-perturbed local models to extract singular value vectors as compact representations of both model structure and privacy noise. The server then clusters clients and identifies a consensus subspace for projecting models with varying noise levels, ultimately aggregating the global model through a residual-aware mechanism. This method not only effectively guides the aggregation of client personalized differential privacy but also reduces communication overhead. Extensive experiments demonstrate the model's effectiveness. Additionally, we provide theoretical proof of the privacy and convergence of FedSPA. Experimental results also showcase its superior performance over personalized DP-FL baselines.
Tianchi Liao, Xiaojun Deng, Lele Fu et al.· Proceedings of the 32nd ACM...· 0 citations
Federated Learning (FL) avoids centralizing raw data, but server-side access to per-client updates still creates a significant privacy risk because gradients can leak sensitive information through inversion and related attacks. A common defense is client-level Differential Privacy (DP), which reduces attack fidelity by perturbing individual updates, but often at the cost of degraded predictive utility, especially in heterogeneous settings. In this paper, we study a different design choice: aggregate-only observability, where the server learns only an aggregate update rather than the individual client contributions that compose it. We develop a unified evaluation framework comparing baseline FL, client-level DP-FL, an idealized sum-only aggregation mechanism (hide_sum), and a quantum-inspired aggregate-observation primitive (qsum) based on finite-shot GHZ-style phase estimation. Experiments on Texas PUDF and MIMIC-IV show that the aggregate-only regimes preserve competitive predictive utility relative to baseline FL while removing the direct per-client observation channel used by standard server-side inversion attacks. We also analyze utility sensitivity to shot budgets and the resource scaling of the quantum-inspired primitive. The results suggest that privacy–utility tradeoffs in FL depend not only on how updates are perturbed, but also on what the server is allowed to observe. Aggregate-only observability therefore provides a useful mechanism-level complement to noise-based privacy defenses, while also motivating hybrid designs to address residual leakage from aggregate updates.
Clifford N. Jones, Md Nahid Hasan, S. Wagle et al.· International Conference on...· 0 citations
Differentially private federated learning must balance privacy protection against model accuracy and training efficiency. Static gradient clipping applies a fixed threshold throughout training and across model layers, which can cause excessive clipping when the threshold is too small or unnecessarily large noise when it is too large. This paper presents DDP-SA-adaptive, an adaptive gradient clipping and noise adding mechanism for differentially private federated learning with secure aggregation. At each communication round, every client determines a separate clipping threshold for each model layer from the median of its per-sample gradient norms. The resulting layer-wise thresholds adapt to the evolving gradient distributions and calibrate the Laplace noise added before the updates are encoded and secret-shared among intermediate aggregation servers. We evaluate the proposed mechanism on a federated regression task in terms of efficiency, accuracy, privacy, convergence, clipping norm, and noise magnitude. Compared with the static DDP-SA baseline, DDP-SA-adaptive reduces the number of communication rounds by 6.81%, total training time by 19.21%, and average per-round training time by 13.33%, leading to improved training efficiency. It also reduces test loss by 98.74% and increases test R2 by 3.41%, leading to improved model accuracy. To attain R2 = 0.99, the adaptive mechanism operates with a privacy budget of approximately epsilon = 0.1, compared with epsilon = 0.4 for static DDP-SA, thus providing stronger privacy protection and achieving stronger privacy guarantees. These results demonstrate that round-wise, layer-wise adaptation can improve the privacy-accuracy-efficiency trade-off of differentially private federated learning.
Wenjing Wei, Alla Jammine, F. Nait-Abdesselam· 0 citations
Split Federated Fine-tuning (SFF) is a promising paradigm for scaling Large Language Models (LLMs) by partitioning model depth between resource-constrained clients and a centralized server. While system incentives for throughput and privacy favor deep partitions, the impact of such configurations on model utility remains poorly understood. In this work, we identify and characterize the Depth-Performance Dilemma: the regime that maximizes system efficiency is precisely where fine-tuning quality collapses. Through a comprehensive audit across four model scales (GPT-2 to Llama-3-8B) and diverse benchmarks, we demonstrate that deeper partitions provide monotonic gains in throughput and privacy at the cost of catastrophic performance plateaus. We evaluate a suite of state-of-the-art federated adapter aggregation methods including AVG, STACK, SVD, and FREEZE, revealing that while these techniques are effective in standard Federated Learning, they fail to mitigate the artifacts unique to split architectures. Finally, we provide a mechanistic diagnosis for this failure, tracing the collapse to the near-isometric topology of Transformers, which allows aggregation noise to propagate without attenuation until it triggers Attention Collapse in the server partition. Our findings challenge the prevailing assumption that partition depth is a utility-neutral tuning knob and provide a structural foundation for stable distributed LLM fine-tuning.
Federated learning (FL) is a popular distributed learning framework where multiple clients perform local training and a server aggregates the locally updated models. FL enables decentralized training while preserving the privacy of clients'datasets. However, non-independent and identically distributed (non-IID) or noisy datasets can lead to low model accuracy or high convergence latency. Precluding these clients through client selection may mitigate the problem, but heavily biased client selections may also degrade the learning performance. In this study, we first experimentally measure the impact of non-IID data (including skews in data quantity and label distribution), noisy data, and fairness in client selection on model accuracy and convergence. We then propose a privacy-preserving scoring method to assess each client's contribution in FL, with experiments conducted to demonstrate the effectiveness of the proposed assessment.
HeteroFL-LoRA is proposed, a federated framework that enables LoRA fine-tuning across heterogeneous LFMs and introduces a Singular Matrix–Guided Subspace Projection that employs singular matrices to achieve cross-subspace mapping, enabling heterogeneous clients to aggregate their LoRA updates in a unified representation space.
Zhuojia Wu, Qi Zhang, Xuerong Zhao et al.· Proceedings of the 32nd ACM...· 0 citations
