Experimental results show that routing-aware collaboration consistently improves personalized performance compared to conventional federated averaging and local training, while maintaining the same communication cost, and shows that client-centric and expert-centric clustering provides an effective and scalable approach for personalized federated instruction fine-tuning of sparse MoE LLMs.
Abstract
Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing. Recent Mixture-of-Experts (MoE) LLMs are particularly attractive for federated learning because their sparse activation reduces computation and communication while scaling model capacity. However, existing federated MoE methods primarily focus on parameter aggregation and personalization, overlooking the routing behavior of MoE models as a source of information for client collaboration. Under heterogeneous instruction distributions, indiscriminate aggregation can lead to negative transfer, highlighting the need to identify which clients should collaborate during federated optimization. We propose ClientMorpher, a routing-aware, personalized federated instruction fine-tuning framework that leverages routing signatures from pretrained MoE models to organize client collaboration prior to aggregation. We investigate two complementary clustering strategies: ClientMorpher-C, which directly clusters clients using expert activation profiles, and ClientMorpher-E, which first clusters experts based on their cross-client usage signatures and then derives client collaboration groups. We evaluate ClientMorpher for federated instruction fine-tuning on the Databricks Dolly-15K dataset, using pathological and Dirichlet-based heterogeneous client distributions across multiple instruction-following tasks. Experimental results show that routing-aware collaboration consistently improves personalized performance compared to conventional federated averaging and local training, while maintaining the same communication cost. Furthermore, our study shows that client-centric and expert-centric clustering provides an effective and scalable approach for personalized federated instruction fine-tuning of sparse MoE LLMs.
Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples. However, task heterogeneity across clients can cause cross-task interference and gradient conflicts during aggregation. Federated MoE-LoRA addresses this challenge through specialized LoRA experts and conditional routing. Yet existing methods typically specialize at client granularity, implicitly assuming task-coherent clients. Our core insight is that experts need purity, namely pattern-coherent updates that preserve specialization, whereas routers need contrast, namely mixed-task observations that support expert comparison. We propose FedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization to meet these two requirements. FedWeave uses unsupervised prototype discovery to form local buckets and align them across clients, enabling prototype-level expert aggregation while retaining mixed-task client trajectories for router training. At inference, FedWeave performs sparse inference with one active expert while preserving nearly all soft-routing performance. Our theoretical analysis explains why asymmetric aggregation is advantageous: it controls expert convergence in stationarity through off-pattern contamination, identifies the consensus error induced by fragmented router trajectories, and bounds sparse-inference risk. On a heterogeneous multi-task benchmark with mainstream LLM backbones, FedWeave consistently outperforms strong baselines, while ablations verify the effectiveness of our design.
Donghang Duan, Xu Zheng, Lizong Zhang et al.· 0 citations
Federated Learning (FL) enables collaborative model training across distributed clients without sharing raw data, but its performance degrades significantly under heterogeneous data distributions. While Clustered Federated Learning (CFL) mitigates heterogeneity by grouping similar clients, most existing methods primarily address label distribution skew and overlook feature distribution skew, where clients share labels but differ systematically in input characteristics. In this paper, we reveal a key observation: under feature distribution skew, different network layers exhibit highly unequal sensitivity, with early layers encoding strongly client-specific feature patterns that dominate aggregation behavior. Motivated by this, we propose SensCluster, a novel sensitivity-aware CFL framework that constructs compact client representations by selecting parameters that are most responsive to local feature distributions. By leveraging Fisher information, SensCluster identifies and retains only the most influential parameters, constructing sensitivity-aware representations for clustering while reducing interference from irrelevant features. Extensive experiments on both image (FEMNIST, CIFAR10-c, CINIC10-c, EuroSAT) and time-series (HAR) benchmarks demonstrate that SensCluster consistently outperforms state-of-the-art CFL methods across diverse feature skew scenarios.
Jiaqi Wang, Tobias Schlagenhauf, Setareh Maghsudi· Proceedings of the 32nd ACM...· 0 citations
FeLiX is presented, an FL orchestration framework that minimizes wall-clock time-to-target accuracy on live interaction streams and achieves near-oracular performance in real-world settings.
Dhruv Garg, Neha Lakhani, Debopam Sanyal et al.· 0 citations
Heterogeneous federated learning leads to system and data differentials that cause stragglers to either be a bottleneck to synchronous optimization or create representation bias in asynchronous contexts. Although current approaches deal with staleness or buffering independently, their approach does not ensure fast clients do not take over the global model. The proposed framework Straggler-Aware Asynchronous Federated Learning (SAFL), that re-defines the stragglers as structured subjects rather than outliers. SAFL employs temporal exponentially weighted moving average signature of client costs and costs model updates by clustering costs in time-constrained per-cluster buffers. An innovative fairness-sensitive aggregation scheme then balances the participation through frequency compensation and damping on staleness. The results of the experiment indicate that SAFL achieves a 75% accuracy in 620 seconds, 27% higher than the state-of-the-art Federated Asynchronous Mobile Update (FedASMU) and increases the fairness index by 0.52 to 0.87. SAFL has a scalable, fair approach to the regulation of heterogeneous clusters, which means they can be used to ensure almost equal contribution in regulated settings such as financial and healthcare analytics.
S. Babalola· 2026 7th International Confe...· 0 citations