Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 5547-5558· 0 citations· 14 references
TL;DR
HeteroFL-LoRA is proposed, a federated framework that enables LoRA fine-tuning across heterogeneous LFMs and introduces a Singular Matrix–Guided Subspace Projection that employs singular matrices to achieve cross-subspace mapping, enabling heterogeneous clients to aggregate their LoRA updates in a unified representation space.
Abstract
Recent studies have integrated Low-Rank Adaptation (LoRA) into federated learning to enable parameter-efficient fine-tuning of Large Foundation Models (LFMs) across data silos, supporting responsible and privacy-preserving collaboration in distributed environments. However, these studies assume that all clients share the same LFM, so each client fine-tunes LoRA on private data using identical pre-trained weights and the server directly aggregates the resulting LoRA updates for collaborative training. The ideal assumption rarely holds in real-world deployments, where clients typically possess heterogeneous base LFMs. Because LoRA updates are anchored to model-specific pre-trained weights, direct aggregation is not only invalid but often counterproductive, leading to sharply degraded collaborative performance. To address this issue, we propose HeteroFL-LoRA, a federated framework that enables LoRA fine-tuning across heterogeneous LFMs. We express LoRA updates through singular value adjustments, activating task-specific knowledge in an parameter-efficient manner while decoupling them from model-specific pre-trained weights. Then, we introduce a Singular Matrix–Guided Subspace Projection that employs singular matrices to achieve cross-subspace mapping, enabling heterogeneous clients to aggregate their LoRA updates in a unified representation space. In this way, task-specific knowledge can be effectively transferred and shared across different LFMs. Experimental results demonstrate that HeteroFL-LoRA achieves effective and robust collaboration across heterogeneous LFMs while maintaining advantages in both training and communication efficiency.
Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-$k$ selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.
Lei Wang, Jieming Bian, Letian Zhang et al.· 0 citations
This work proposes SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator, and introduces a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting.
Yue Xia, Tayyebeh Jahani-Nezhad, Mayank Bakshi et al.· 1 citation
i-FedLoRA provides privacy guarantees, improves model accuracy by up to 3.8%, and expedites training by 1.37-2.23×, and facilitates heterogeneous LoRA aggregation that selectively prioritizes high-confidence knowledge to filter DP-induced noise, thereby achieving robust knowledge transfer.
Nan Yan, Yuqing Li, Xiong Wang et al.· Proceedings of the 32nd ACM...· 0 citations
Dynamic Subspace Boosting (Dysco), a plug-in method that allocates client-specific LoRA subspaces in a federated and dynamic manner, and proves that Dysco's server-fixed merged subspaces yield a tighter upper bound on this error.
Haobo Zhang, Jiankun Wang, Suraj Rajendran et al.· 0 citations
Low-Rank Adaptation (LoRA) enables communication-efficient federated fine-tuning of pretrained language models. However, integrating differential privacy (DP) into federated LoRA remains challenging: independently perturbing and aggregating its two low-rank matrices can cause aggregation mismatch and the quadratic noise term. Existing methods mitigate these issues by freezing one low-rank matrix but still rely on Euclidean aggregation, which is basis-dependent and may distort the global update. To address this limitation, we propose FedGSA, a geometry-consistent aggregation framework for differentially private federated LoRA. FedGSA represents each privatized client update as a basis-invariant subspace on the Grassmann manifold. In each communication round, clients extract low-dimensional subspaces capturing dominant update directions and encode them as projection matrices. The server aggregates these representations to estimate a geometry-consistent global update subspace and reconstructs the global LoRA factors within it, reducing distortion caused by basis misalignment, privacy noise, and heterogeneous client updates. We prove that FedGSA incurs no additional privacy loss beyond client-side DP training and establish its convergence under standard assumptions. Experiments on four GLUE tasks and a language generation benchmark demonstrate consistent improvements across privacy budgets and degrees of data heterogeneity. In particular, FedGSA improves average accuracy over the strongest baseline by 2.17% and 2.27% under $\epsilon=6$ and $\epsilon=3$, respectively.
Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a $6.82$ percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.
Juseok Jeon, Ramy E. Ali, Doyun Kwon et al.· 0 citations
