CFLoRA is presented, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round, and eliminates bilinear terms in matrix multiplications, making federated aggregation exact.
Abstract
Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present \texttt{CFLoRA}, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates \texttt{CFLoRA} achieves $\mathcal{O}(1/\sqrt{T})$ convergence rate of the \textit{original} LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that \texttt{CFLoRA} achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.
Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental “aggregation dilemma” between the accurate Sum-of-Products (SoP) and the commun...
Han Zou, C. Zhang, Yu-Zhi Yang et al.· IEEE Journal on Selected Are...· 0 citations
This work proposes SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator, and introduces a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting.
Yue Xia, Tayyebeh Jahani-Nezhad, Mayank Bakshi et al.· 2 citations
FedRoRA (Federated Rank-wise Personalized LoRA) is proposed, a novel framework that enables fine-grained personalization within rank-heterogeneous federations and consistently outperforms state-of-the-art methods.
Lei Wang, Jieming Bian, Letian Zhang et al.· 0 citations
FedLAFP is proposed, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch, and consistently outperforms local-only and federated LoRA baselines.
Meng-Jun Yi, Huai-An Gu, Yi-Hao Ai et al.· arXiv.org· 0 citations
FedPA-LoRA is proposed, a product-aligned federated LoRA framework that jointly addresses limitations and provably converges under both homogeneous and heterogeneous client ranks, with up to a $6.82$ percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.
Juseok Jeon, Ramy E. Ali, Doyun Kwon et al.· 1 citation
SplitLite is proposed, a communication-efficient split federated LoRA fine-tuning method that exploits the low effective rank structure of consecutive-epoch activation and gradient residuals, thereby significantly reducing both activation uplink and gradient downlink traffic.
Tao Li, Yu-Lin Tang, Qi Guo et al.· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026