Skip to content

HO-FL: Hybrid-Order Federated Learning for Heterogeneous Edge Devices

Sep 2026 · 0 citations · 25 references
Computer Science

TL;DR

HO-FL is introduced, a hybrid-order FL framework that trains a model's bottom segment with ZO optimization and its top segment with FO optimization, and shows that hybrid-order local training can retain much of the full-FO performance with substantially lower client memory requirements.

Abstract

Federated learning (FL) on memory-constrained edge devices faces a dilemma: first-order (FO) optimization (i.e., backpropagation) demands substantial memory, whereas zeroth-order (ZO) optimization suffers from severe convergence slowdown. To resolve this dilemma, we introduce HO-FL, a hybrid-order FL framework that trains a model's bottom segment with ZO optimization and its top segment with FO optimization. Each device can flexibly select its order boundary according to its memory budget while participating in the training of the same global model. Moreover, our convergence analysis reveals a new, fundamental trade-off: clients with larger FO-trained segments can provide more accurate updates, but favoring them can underrepresent other clients'data. We connect this trade-off to the bias and variance of actual multi-step local updates, yielding a sampling optimization problem and a practical dimension-aware approximation with direct model averaging. Experiments on language tasks examine task performance, client memory, and sampling under data heterogeneity. The results show that hybrid-order local training can retain much of the full-FO performance with substantially lower client memory requirements. Our code is available at https://github.com/HKU-WILL-Lab/HO-FL.

View source

Similar papers

#machine learning Preprint Sep 2026

FANS: Federated Adaptive Network Search Learning for Heterogeneous Devices

The Federated Parallel Scaling (FPS) algorithm is proposed, which jointly trains multiple sampled subnetworks in parallel with self-distillation so that larger sampled subnetworks can supervise smaller ones during local updates.

Jia-Xin Zhang, Xing-Wei Wang, Bo Yi et al. · 0 citations
Conference Open access Aug 2026

ELSA-FL: Energy and Load Serving Aware Federated Learning

Federated Learning (FL) enables collaborative model training across distributed devices. A major concern in FL is how to operate it smoothly in resource-constrained environments, where this process must perform under strict operational constraints, such as cost or energy budgets. An often-overlooked aspect is the effec...

Anna Lackinger, P. Frangoudis, Andrea Morichetta et al. · 0 citations
Open access Dec 2024

Fed-ZOE: Communication-Efficient Over-the-Air Federated Learning via Zeroth-Order Estimation

As 6G and beyond networks grow increasingly complex and interconnected, federated learning (FL) emerges as an indispensable paradigm for securely and efficiently leveraging decentralized edge data for AI. By virtue of the superposition property of communication signals, over-the-air FL (OtA-FL) achieves constant commun...

Jonggyu Jang, Hyeonsu Lyu, D. J. Love et al. · 4 citations
#machine learning Open access Oct 2026

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization

Federated Learning (FL) enables privacy-preserving fine-tuning of Large Language Models (LLMs), yet the massive communication overhead remains a critical bottleneck. Furthermore, applying Low-Rank Adaptation (LoRA) in FL faces a fundamental “aggregation dilemma” between the accurate Sum-of-Products (SoP) and the commun...

Han Zou, C. Zhang, Yu-Zhi Yang et al. · 0 citations
Open access Aug 2026

EA-AQF: energy aware adaptive quantization and freezing in federated learning

Energy-Aware Adaptive Quantization and Freezing (EA-AQF), a unified framework that co-optimizes communication and computation, is presented, a unified framework that co-optimizes communication and computation and maintains robust convergence in highly heterogeneous tasks.

Farwa Ikram, Sadi Alawadi, Dipanwita Thakur et al. · 0 citations
Review Open access Sep 2026

Task-Oriented Framework for Communication-Efficient Federated Learning: From Isolated Optimization to Holistic Synergy

This article presents an autonomous driving case study with two complementary experiments: a task-oriented client scheduling strategy that improves object detection accuracy under the same communication time budget, and a joint quantization-bandwidth optimization that further reduces total training time under dynamic n...

Fu-Qiang Pan, Yan Liu, Er-Wu Liu et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.