HO-FL is introduced, a hybrid-order FL framework that trains a model's bottom segment with ZO optimization and its top segment with FO optimization, and shows that hybrid-order local training can retain much of the full-FO performance with substantially lower client memory requirements.
Abstract
Federated learning (FL) on memory-constrained edge devices faces a dilemma: first-order (FO) optimization (i.e., backpropagation) demands substantial memory, whereas zeroth-order (ZO) optimization suffers from severe convergence slowdown. To resolve this dilemma, we introduce HO-FL, a hybrid-order FL framework that trains a model's bottom segment with ZO optimization and its top segment with FO optimization. Each device can flexibly select its order boundary according to its memory budget while participating in the training of the same global model. Moreover, our convergence analysis reveals a new, fundamental trade-off: clients with larger FO-trained segments can provide more accurate updates, but favoring them can underrepresent other clients'data. We connect this trade-off to the bias and variance of actual multi-step local updates, yielding a sampling optimization problem and a practical dimension-aware approximation with direct model averaging. Experiments on language tasks examine task performance, client memory, and sampling under data heterogeneity. The results show that hybrid-order local training can retain much of the full-FO performance with substantially lower client memory requirements. Our code is available at https://github.com/HKU-WILL-Lab/HO-FL.
The Federated Parallel Scaling (FPS) algorithm is proposed, which jointly trains multiple sampled subnetworks in parallel with self-distillation so that larger sampled subnetworks can supervise smaller ones during local updates.
Jia-Xin Zhang, Xing-Wei Wang, Bo Yi et al.· 0 citations
Federated Learning (FL) enables collaborative model training across distributed devices. A major concern in FL is how to operate it smoothly in resource-constrained environments, where this process must perform under strict operational constraints, such as cost or energy budgets. An often-overlooked aspect is the effec...
Anna Lackinger, P. Frangoudis, Andrea Morichetta et al.· 2026 International Conferenc...· 0 citations
As 6G and beyond networks grow increasingly complex and interconnected, federated learning (FL) emerges as an indispensable paradigm for securely and efficiently leveraging decentralized edge data for AI. By virtue of the superposition property of communication signals, over-the-air FL (OtA-FL) achieves constant commun...
Jonggyu Jang, Hyeonsu Lyu, D. J. Love et al.· IEEE Journal on Selected Are...· 4 citations
Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental “aggregation dilemma” between the accurate Sum-of-Products (SoP) and the commun...
Han Zou, C. Zhang, Yu-Zhi Yang et al.· IEEE Journal on Selected Are...· 0 citations
Energy-Aware Adaptive Quantization and Freezing (EA-AQF), a unified framework that co-optimizes communication and computation, is presented, a unified framework that co-optimizes communication and computation and maintains robust convergence in highly heterogeneous tasks.
This article presents an autonomous driving case study with two complementary experiments: a task-oriented client scheduling strategy that improves object detection accuracy under the same communication time budget, and a joint quantization-bandwidth optimization that further reduces total training time under dynamic n...
Fu-Qiang Pan, Yan Liu, Er-Wu Liu et al.· IEEE Internet of Things Maga...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.