Skip to content

What Should Federated LoRA Share? FedSAIL via Input-aware Subspace Alignment

Sep 2026 · 0 citations · 19 references
Computer Science

TL;DR

This work introduces an input-aware action matrix that weights the adapter update by the second-moment statistics of local layer inputs and proposes Federated Subspace-Guided Action-Informed Learning (FedSAIL), which improves predictive performance over competing federated LoRA methods while reducing communication cost significantly.

Abstract

Federated low-rank adaptation (LoRA) requires identifying an update structure that is shared across heterogeneous clients. Prior work reports strong similarity among trained LoRA projection matrices across clients; however, such agreement may be largely induced by common initialization and collapses toward random overlap under independent initialization. More crucially, relying solely on parameter similarity inherently ignores the influence of local input regime. To uncover a more robust shared structure, we introduce an input-aware action matrix that weights the adapter update by the second-moment statistics of local layer inputs. Empirically, while parameter similarity vanishes, the leading right singular directions of this action matrix remain strongly aligned across clients. This shared geometry preserves task-conditioned differences and naturally varies across network depths. Motivated by these findings, we propose Federated Subspace-Guided Action-Informed Learning (FedSAIL). Instead of averaging weights, FedSAIL estimates a shared action subspace to regularize local training while preserving client-specific coefficients. Across several benchmarks, our approach consistently improves predictive performance over competing federated LoRA methods while reducing communication cost significantly.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create...

Jun-Kang Liu · 0 citations
#artificial intelligence Preprint Sep 2026

FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon

FedSubMuon is proposed, a communication-efficient federated Muon fine-tuning method that optimizes compact coefficient matrices within shared structured subspaces that keeps Muon on a single matrix-valued trainable object, while reducing the client upload to compact coefficient matrices.

Shao-Long Chen, Youming Tao, Shu-Zhen Chen et al. · 0 citations
Preprint Aug 2026

GCA: Global Centroid Alignment in Federated Learning

Global Centroid Alignment (GCA), a latent-code-mediated FL protocol that coordinates clients without transmitting AE parameters or gradients, achieves extraction defense comparable to DP-FedAvg, remains effective when DP-FedAvg does not reduce target resemblance, and lowers per-round communication.

Jong-Ik Park, Harry H. Jiang, Logan Blakely et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.