This work introduces an input-aware action matrix that weights the adapter update by the second-moment statistics of local layer inputs and proposes Federated Subspace-Guided Action-Informed Learning (FedSAIL), which improves predictive performance over competing federated LoRA methods while reducing communication cost significantly.
Abstract
Federated low-rank adaptation (LoRA) requires identifying an update structure that is shared across heterogeneous clients. Prior work reports strong similarity among trained LoRA projection matrices across clients; however, such agreement may be largely induced by common initialization and collapses toward random overlap under independent initialization. More crucially, relying solely on parameter similarity inherently ignores the influence of local input regime. To uncover a more robust shared structure, we introduce an input-aware action matrix that weights the adapter update by the second-moment statistics of local layer inputs. Empirically, while parameter similarity vanishes, the leading right singular directions of this action matrix remain strongly aligned across clients. This shared geometry preserves task-conditioned differences and naturally varies across network depths. Motivated by these findings, we propose Federated Subspace-Guided Action-Informed Learning (FedSAIL). Instead of averaging weights, FedSAIL estimates a shared action subspace to regularize local training while preserving client-specific coefficients. Across several benchmarks, our approach consistently improves predictive performance over competing federated LoRA methods while reducing communication cost significantly.
Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create...
FedSubMuon is proposed, a communication-efficient federated Muon fine-tuning method that optimizes compact coefficient matrices within shared structured subspaces that keeps Muon on a single matrix-valued trainable object, while reducing the client upload to compact coefficient matrices.
Shao-Long Chen, Youming Tao, Shu-Zhen Chen et al.· 0 citations
FedRoRA (Federated Rank-wise Personalized LoRA) is proposed, a novel framework that enables fine-grained personalization within rank-heterogeneous federations and consistently outperforms state-of-the-art methods.
Lei Wang, Jieming Bian, Letian Zhang et al.· 0 citations
FedLAFP is proposed, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch, and consistently outperforms local-only and federated LoRA baselines.
Meng-Jun Yi, Huai-An Gu, Yi-Hao Ai et al.· arXiv.org· 0 citations
CFLoRA is presented, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round, and eliminates bilinear terms in matrix multiplications, making federated aggregation exact.
Global Centroid Alignment (GCA), a latent-code-mediated FL protocol that coordinates clients without transmitting AE parameters or gradients, achieves extraction defense comparable to DP-FedAvg, remains effective when DP-FedAvg does not reduce target resemblance, and lowers per-round communication.
Jong-Ik Park, Harry H. Jiang, Logan Blakely et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026