Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 4463-4474· 0 citations· 16 references
TL;DR
HeimdaLLM is a cloud-assisted federated fine-tuning framework that combines ZOO with Gradient Rectification (ZGR) and reduces memory footprint for client devices, achieves up to 8.8× faster convergence than the baselines, and improves accuracy by up to 10% over state-of-the-art ZOO methods.
Abstract
Large Language Models (LLMs) have achieved remarkable success in NLP tasks, but fine-tuning them on resource-constrained mobile devices remains challenging due to prohibitive memory and computation requirements. Federated Learning (FL) enables privacy-preserving distributed fine-tuning, yet conventional approaches, including full fine-tuning and parameter-efficient fine-tuning (PEFT), either rely on backpropagation, which is memory-intensive, or zeroth-order optimization (ZOO), which suffers from high variance and directional bias, leading to slow convergence and degrading accuracy. To address these limitations, we propose HeimdaLLM, a cloud-assisted federated fine-tuning framework that combines ZOO with Gradient Rectification (ZGR). In HeimdaLLM, FL clients perform ZOO updates on local domain datasets, while the assisted cloud conducts BP-based fine-tuning on datasets from the same domain and periodically transmits guided gradient direction (GGD) data to rectify ZOO process. To reduce client memory usage, computation, and transmission overhead, we introduce two system-level controllers: the System-level Pipeline Controller (SPC) and Data Transmission Controller (DTC), which pipeline model layer-wise transmission and computation of cloud and clients, and adaptively compress GGD data. Experiments on multiple models and NLP datasets show that HeimdaLLM reduces memory footprint for client devices by up to 86.37%, achieves up to 8.8× faster convergence than the baselines, and improves accuracy by up to 10% over state-of-the-art ZOO methods.
Federated fine-tuning is bottlenecked by communication: FedAvg and pseudo-gradient schemes transmit a payload that scales with the model, and gradient compression shrinks it by only a constant factor. We take a different lever. Mapping networks generate a network's weights from a small trainable latent through a frozen affine projection; because the map is shared and affine, averaging latents is exactly averaging the generated weights. We turn this into a practical low-bandwidth federated channel with two changes: a low-rank, seed-regenerable factorisation of the projection (cutting generator memory from ~80 GB to ~10 MB), and a delta formulation $\theta = \theta^{\mathrm{pre}} + U V^{\top} z$ that learns an additive correction around a shared centrally-pretrained base -- federated fine-tuning, which is what makes the method work at scale. A frozen orthogonal classifier head further removes the head from the payload while improving accuracy. On CIFAR-100 with ResNet-18+GroupNorm, our method (FLITE, Federated Low-rank Iterative Training Engine) communicates 1,280 floats (~5 KB) per client per round -- an 8718x reduction -- and reaches 74.67%, within ~0.5 pp of full-weight FedAvg. The averaging identity holds to floating-point precision ($6 \times 10^{-8}$); the method sits one to two orders of magnitude below PowerSGD and top-k on the bandwidth-accuracy Pareto; it matches or exceeds full-weight FedAvg under strong non-IID skew. int4 latents reach 648 bytes per round at unchanged accuracy, whereas int4 full-weight FedAvg collapses to chance.
Energy-Aware Adaptive Quantization and Freezing (EA-AQF), a unified framework that co-optimizes communication and computation, is presented, a unified framework that co-optimizes communication and computation and maintains robust convergence in highly heterogeneous tasks.
TRISHUL is proposed, a spectral-control framework for robust federated PEFT that follows the FL no-raw-data-sharing setting but does not itself provide formal privacy guarantees, and improves convergence, stability, and final performance over federated LoRA baselines.
SplitLite is proposed, a communication-efficient split federated LoRA fine-tuning method that exploits the low effective rank structure of consecutive-epoch activation and gradient residuals, thereby significantly reducing both activation uplink and gradient downlink traffic.
FeDiSyn is proposed, a unified framework that holistically considers the interplay between pre-training and FedFT to minimize the overall LVM training time and introduces a scaling law for FedFT pre-training to determine the optimal number of synthetic images, balancing pre-training benefit against generation/pre-training cost.
Qianpiao Ma, Xiaozhu Song, Junlong Zhou et al.· 0 citations
Experimental results show that FedOPAL not only significantly outperforms the original analytical methods on several benchmarks, but also achieves accuracy comparable to state-of-the-art iterative methods while maintaining zero server-side training costs, providing a new engineering paradigm for efficient collaboration of large models on the edge.
Lingyu Qiu, Daniela Annunziata, Stefano Izzo et al.· 2026 2nd International Confe...· 0 citations