Skip to content
Log in
Open access

H2-LBM: A Hierarchical Hybrid Deep Reinforcement Learning Framework for L7 Load Balancing and Global Traffic Scheduling in Multi-Cloud LLM Serving

Chengde Xu Chongye Wang Zeyu Li
Jun 2026 · Fundamental Scientific Reports in Multidisciplinary Areas · 0 citations

TL;DR

Experimental results consistently validate the effectiveness of H2-LBM in improving latency stability and system efficiency for large-scale LLM inference services.

Abstract

Large Language Model (LLM) inference services are increasingly deployed across multi-cloud infrastructures to support latency-sensitive applications such as conversational AI, intelligent search, and real-time analytics. However, the coexistence of heterogeneous computing resources, geographically distributed data centers, and highly dynamic request patterns poses significant challenges to efficient L7 load balancing and global traffic scheduling. In particular, traditional rule-based or heuristic-driven approaches fail to capture the semantic variability of LLM workloads, including token length diversity and non-linear inference latency, leading to suboptimal resource utilization and degraded tail latency performance. To address these issues, this paper proposes H2-LBM, a hierarchical hybrid deep reinforcement learning framework designed for multi-cloud LLM serving environments. The framework decomposes the scheduling problem into two coordinated levels: a global scheduler based on Proximal Policy Optimization (PPO) for cross-cloud traffic allocation, and a local scheduler based on Double Dueling Deep Q-Network (D3QN) for fine-grained L7 request dispatching. By incorporating semantic-aware state representations and a multi-objective reward function that jointly optimizes latency, throughput, and resource efficiency, H2-LBM enables adaptive and scalable decision-making under dynamic workloads. Experimental results on a multi-cloud Kubernetes testbed show that H2-LBM reduces P99 latency by 21.5%–29.2% compared with strong baselines such as Kubernetes HPA across moderate and burst workloads. Specifically, P99 latency decreases from 360 ms to 255 ms under moderate load and from 650 ms to 510 ms under high load. These results consistently validate the effectiveness of H2-LBM in improving latency stability and system efficiency for large-scale LLM inference services.

DOI Semantic Scholar
Read PDF

Similar papers

Open access Jun 2026

DEEP REINFORCEMENT LEARNING-BASED INTELLIGENT TASK SCHEDULING FRAMEWORK FOR CLOUD DISTRIBUTED SYSTEMS

This paper proposes an innovative Deep Reinforcement Learning-based Intelligent Task Scheduling Framework (DRITS) designed to optimize task allocation and resource utilization in cloud distributed systems and establishes DRL-based intelligent scheduling as a promising solution for next-generation cloud computing infrastructure management.

Tileemat Ashour Aletiri · 0 citations
Open access Jul 2026

STQ-Scheduler: A Secure and Throughput-Aware Deep Reinforcement Learning Framework for QoE-Driven Resource Scheduling in Distributed Video Streaming Systems

STQ-Scheduler is proposed, a secure and throughput-aware deep reinforcement learning framework that integrates high-throughput data processing, Transformer-based QoE prediction, and Proximal Policy Optimization-based resource scheduling to ensure data quality and prevent data processing from becoming a bottleneck in distributed training.

Yi-Chun Chang, Min-Wei Jiang · 0 citations
2026

Hierarchical Meta-Graph Reinforcement Learning for Collaborative GenAI Model Caching and Inference Scheduling

Enabling collaborative generative AI (GenAI) inference at the network edge is challenging due to limited caching capacity, heterogeneous computing resources, and highly dynamic, latency-sensitive service demands. In this paper, we investigate the joint optimization of GenAI model caching, inference offloading, and resource allocation in a collaborative cloud–edge–end architecture. To address the strong coupling between long-term caching decisions and short-term scheduling dynamics, we propose a Hierarchical Meta-Graph Reinforcement Learning framework, termed HMGRL. Specifically, a heat-greedy model caching strategy is developed to capture time-varying model popularity and to reduce switching overhead on a slow timescale, while a graph-enhanced dueling deep reinforcement learning algorithm with prioritized experience replay enables topology-aware collaborative inference offloading and resource allocation on a fast timescale. Extensive simulations demonstrate that HMGRL consistently outperforms representative baselines in terms of system utility, cache and computing-resource utilization, convergence stability, and performance robustness. These results validate the effectiveness of the proposed hierarchical learning framework for practical GenAI applications at the network edge.

Liang Zhao, Jing Wei, Huan Zhou et al. · 0 citations
Preprint Jun 2026

Multi-Timescale Latent-Action DRL for Joint Optimization in Edge-Cloud Networks

A two-timescale multi-layer deep reinforcement learning framework with a latent action space (2T-MDRL-LA) to jointly optimize service placement, user association, computational delegation, task offloading, and user transmit power and achieves near-optimal performance compared to branch-and-bound solutions.

V. Son, Van-Dinh Nguyen, Ngoc Hung Nguyen et al. · 0 citations