Skip to content

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Sep 2026 · arXiv.org · Vol abs/2609.37033 · 0 citations · 29 references
Computer Science

TL;DR

FedLAFP is proposed, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch, and consistently outperforms local-only and federated LoRA baselines.

Abstract

Federated parameter-efficient fine-tuning enables clients to adapt pre-trained models without sharing raw data or communicating the full model, but statistical heterogeneity makes a single global adapter insufficient for personalized prediction. Existing personalized methods typically use the same low-rank structure for both shared and private adaptation, overlooking their distinct requirements for aggregation and personalization. We propose FedLAFP, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch. The shared branch provides an efficient interface for transferring common knowledge, whereas the private branch combines fixed random low-rank bases with learned scaling coefficients to provide expressive client-specific adaptation without additional communication. Client- and layer-specific mixing coefficients jointly fuse the two branches, and only the shared LoRA parameters are exchanged. A controlled linear study supports this role assignment: LoRA yields more aligned client updates and lower aggregation error, while RandLoRA more accurately recovers client-specific residuals. Experiments across four visual recognition benchmarks show that FedLAFP consistently outperforms local-only and federated LoRA baselines, achieving an average personalized accuracy of $86.93\%$ and exceeding the best baseline average by $1.30$ percentage points.

View source

Similar papers

#machine learning Preprint Oct 2026

ORDERS: An Empirical Study of Norm-Rank Aggregation for Personalized Federated Learning

Personalized federated learning combines shared representations with client-specific predictors, but the contribution of a server weighting rule can be obscured by local training and evaluation choices. We study ORDERS, a configuration that combines a shared backbone, a private residual adapter and classifier, geometri...

Koffka Khan · 0 citations
#artificial intelligence Preprint Oct 2026

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create...

Jun-Kang Liu · 0 citations
Preprint Nov 2026

CARE-FL: Federated Fine-Tuning of Multimodal LLMs in Resource-Constrained Edge Networks

Domain-specific adaptation of Multimodal Large Language Models (MLLMs) in privacy-sensitive edge environments requires federated fine-tuning, yet client selection remains challenging due to heterogeneous multimodal data and modality availability, dynamic system conditions, and evolving client utility during training. W...

Tian-Qi Zong, Bao-Sheng Li, Nuo-Cheng Yang et al. · 0 citations
#machine learning Open access Oct 2026

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization

Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental “aggregation dilemma” between the accurate Sum-of-Products (SoP) and the commun...

Han Zou, C. Zhang, Yu-Zhi Yang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.