Skip to content
Log in
Open access

Nested Multi-Agent Reinforcement Learning for Adaptive Resource Management in 6G Network Slicing: A Multi-Timescale Framework With Convergence Guarantees

Abraheem Rashid Faisal Iradat Waseem Iqbal Ikram Syed Khalil Khan Khalid Yahya
2026 · IEEE Open Journal of the Communications Society · Vol 7, pp. 7624-7640 · 0 citations · 39 references
Computer Science

TL;DR

A finite-time convergence result in the two-timescale stochastic approximation framework is established showing that under standard regularity and timescale-separation conditions, Nested-MARL achieves a lower bound for any single-timescale algorithm.

Abstract

Sixth-generation (6G) networks are expected to rely on agentic artificial intelligence for zero-touch, self-managed orchestration of heterogeneous network slices serving enhanced mobile broadband (eMBB), ultra-reliable low-latency communication (URLLC), and massive machine-type communication (mMTC). A central and under-studied challenge for adaptive multi-agent resource management (AMRM) in such settings is multi-timescale non-stationarity: channel fading evolves per time-slot, user demand shifts at the window scale, and service-level agreement (SLA) regimes change at an operational scale. Single-timescale multi-agent reinforcement learning (MARL) algorithms cannot track all three signals cleanly—a learning rate fast enough for the per-slot channel destabilises the coordination structure that governs longer-timescale policies. This paper proposes Nested-MARL, an independent-learner actor-critic algorithm in which each agent’s parameters are partitioned into three groups updated at separated rates <inline-formula> <tex-math notation="LaTeX">$\alpha _{0}\!\ll \!\alpha _{1}\!\ll \!\alpha _{2}$ </tex-math></inline-formula>, with a continuum-memory exponential moving average (EMA) anchoring the slowest group. The design is grounded in the Nested Learning paradigm of Behrouz et al. (2025) and is extended here from single-model continual learning to decentralised multi-agent coordination. We establish a finite-time convergence result in the two-timescale stochastic approximation framework showing that under standard regularity and timescale-separation conditions, Nested-MARL achieves <inline-formula> <tex-math notation="LaTeX">$O(T^{-1/2})$ </tex-math></inline-formula> fast-group convergence vs. an <inline-formula> <tex-math notation="LaTeX">$\Omega (T^{-1/3})$ </tex-math></inline-formula> lower bound for any single-timescale algorithm. An empirical study on a three-agent 6G slicing simulator with continuous multi-timescale drift shows Nested-MARL outperforms independent PPO (IPPO) in mean reward at every drift severity we test (<inline-formula> <tex-math notation="LaTeX">$\kappa \!\in \!\{0.5,1.0,1.5,2.0\}$ </tex-math></inline-formula>) and by + 8.6% in sample efficiency over the first 40 episodes at <inline-formula> <tex-math notation="LaTeX">$\kappa {=}1.5$ </tex-math></inline-formula> (<inline-formula> <tex-math notation="LaTeX">$n{=}10$ </tex-math></inline-formula> seeds, <inline-formula> <tex-math notation="LaTeX">$p\lt 0.05$ </tex-math></inline-formula>). A controlled ablation establishes that stripping timescale separation reduces performance below the IPPO baseline, isolating timescale separation as the causal mechanism. Nested-MARL also reduces policy switching cost by 16.6%, an operationally meaningful benefit for zero-touch orchestration. The complete simulator, agents, and 60 + per-seed training runs are released as open source.

DOI Semantic Scholar
Read PDF

Similar papers

2026

Collaborative Task Offloading in Space Computing Power Network: A World Model-Based Multi-Agent Reinforcement Learning Approach

Low Earth Orbit (LEO) constellations are required to process increasing volumes of heterogeneous tasks from ground networks. Intermittent inter-satellite links, heterogeneous onboard resources, and time-varying traffic loads make collaborative task offloading difficult for static or reactive strategies. Although multi-agent reinforcement learning (MARL) provides an adaptive solution, existing model-free MARL methods often suffer from slow convergence, insufficient foresight, and limited robustness in dynamic satellite environments. To address these challenges, this paper proposes a World Model-Based Multi-Agent Proximal Policy Optimization (WM-MAPPO) framework for space computing power networks. The offloading problem is formulated as a partially observable multi-agent decision-making process, where LEO satellites make decentralized decisions under incomplete local observations. A predictive world model learns latent transition dynamics of network states and provides future context for proactive planning. Meanwhile, a Transformer-based policy architecture captures inter-agent dependencies and supports cooperative scheduling under centralized training and decentralized execution (CTDE). Simulation results show that WM-MAPPO achieves higher task completion ratios, lower average latency, improved energy efficiency, and stronger robustness than model-free MARL baselines, heuristic methods, Lyapunov-based scheduling, and MINLP-inspired optimization.

Yuqi Cong, Zhiwei Wei, Jiarui Chen et al. · 0 citations
Preprint Aug 2026

Multi-Agent Reinforcement Learning for Online Traffic Scheduling in Time-Sensitive Application

This paper proposes a multi-agent reinforcement learning (MARL) framework for TSN scheduling, where each TSN queue is modeled as an autonomous agent and the Heterogeneous-Agent Proximal Policy Optimization (HAPPO) algorithm is employed to explicitly model inter-agent dependencies and jointly optimize service delivery across queues.

Marcos Carvalho, Fatih Temiz, Shavbo Salehi et al. · 0 citations
Preprint Jul 2026

Multi-Agent Reinforcement Learning for SLA-Aware Network Slicing in UAV-Enabled MEC

A predictive multi-agent Reinforcement Learning (RL) framework that proactively maintains SLA stability in UAV-enabled MEC through coordinated trajectory control and computation resource allocation and designs an SLA-aware reward function that explicitly penalizes both violation probability and duration across slices.

M. Farhoudi, Zeinab Sasan, Masoud Shokrnezhad et al. · 0 citations
Open access Jul 2026

MULTI-AGENT REINFORCEMENT LEARNING FOR TASK OFFLOADING AND RESOURCE ALLOCATION IN MEC SYSTEMS

This paper addresses the joint task offloading and resource allocation problem in multi-user MEC systems and proposes a decentralized control framework based on Multi-Agent Reinforcement Learning (MARL), which achieves lower total system cost and faster convergence than the full-local, full-offload, and heuristic baselines.

Youssef Oukissou, Mohamed Amine Meddaoui, Ayoub Belaidi et al. · 0 citations
Open access Jul 2026

Robust Offline Multi-Agent Reinforcement Learning for Latency-Aware SDN Path Control in 6G-Oriented Network Softwarization

Future sixth-generation (6G)-oriented networks require programmable control that can adapt routing to latency and congestion without unsafe online exploration. This study evaluates offline multi-agent deep deterministic policy gradient (MADDPG) with behavior-adjusted training rewards for latency-aware path control in software-defined networking (SDN). Each traffic pair is modeled as an agent selecting one of three retained candidate paths, while centralized critics learn coordinated decisions from topology-specific Ryu–Mininet transition datasets. Nine policies are compared using ten paired seeds on fat-tree, mesh-grid, and WAN-corridors topologies under a deployed utilization–latency weighting of 0.60/0.40, together with flow-completion, latency, congestion, architectural-comparison, sensitivity, robustness, statistical, and controller-overhead analyses. The utilization-aware path heuristic achieves the strongest overall reward ranking. MADDPG is the strongest learned policy on fat-tree, is not significantly outperformed by any evaluated policy on mesh-grid, and remains statistically tied with completion-matched policies on WAN-corridors. Behavior adjustment is topology-dependent rather than uniformly beneficial. The exported policy requires approximately 52μs per joint decision, whereas complete control-loop timing is dominated by network-statistics polling. These results support offline multi-agent SDN control as a competitive, low-overhead option when interpreted jointly with topology structure, flow completion, and strong heuristic baselines.

A. Kyzyrkanov, Y. Nurakhov, Zhenis Otarbay et al. · 0 citations
Jul 2026

Hierarchical Multi-Objective Learning for Context-Aware 5G Ran Slice Resource Allocation

Efficient coexistence of eMBB and URLLC services remains a critical challenge in AI-native Radio Access Networks (RANs). This paper proposes a two-timescale Hierarchical Reward Weighting (HRW) framework based on multiobjective reinforcement learning for context-aware O-RAN slicing under a Constrained Markov Decision Process (CMDP) formulation. The proposed architecture separates long-term policy adaptation from fast-timescale radio scheduling, mitigating the non-stationarity inherent in multiobjective RAN optimization. At the slow layer, a non-realtime RIC rApp exploits a long-term network context and a differentiable Softmax mapping to adapt slice reward preferences. These policies are propagated through the $O$ -RAN control hierarchy to guide downstream scheduling decisions. At the fast layer, decentralized scheduling agents embedded within the Open Distributed Unit (O-DU) MAC layer execute sub-millisecond Physical Resource Block (PRB) allocation and packet preemption, avoiding near-RT RIC transport latency constraints. Evaluated under a multiuser MIMO-OFDMA environment, the proposed framework improves resource utilization by up to 60.8% over static partitioning while maintaining bounded URLLC tail-latency behavior and strict Service Level Agreement (SLA) compliance. The results demonstrate the feasibility of AI-native hierarchical O-RAN control and align with the ITU-T visions for autonomous 6G RAN intelligence.

Charles Ssengonzi, Okuthe P. Kogeda, T. Olwal · 0 citations