Relational Route 6G: A Decentralized Dynamic Optimal Routing Through Multi-Agent Advantage Actor-Critic in Next-Gen Networks

Abstract

– The proliferation of 6G networks creates massive amounts of data and links a multitude of devices, which presents problems for conventional approaches to network management. Dynamic routing in 6G networks is a feature that dynamically selects data paths within a live network, adapting to network conditions like congestion, link quality, user mobility and service demands. Reinforcement Learning (RL) is an emerging approach for dynamic routing of 6G networks. Due to the complexity, single-agent RL model is struggled to optimize routing decisions and maximizing a reward function that represents target network objectives. As the scale of the network grew, it became more tedious to train an individual agent. Therefore, a standard Multi-Agent Reinforcement Learning (MARL) model was adapted for dynamic optimal routing in 6G networks. As more agents are added, the states and actions become large in the standard MARL which takes a long time and a more computation to learn. Therefore, this paper proposes a novel MARL using Decentralized Relational Advantage Actor-Critic (MARL-DR-A2C) for improving dynamic routing of 6G networks. The first step is to forecast network congestion parameters like delay, load and queue length based on past data using the Speed-optimized Long Short-Term Memory (SP-LSTM) network. Then, a Multi-Agent Partially Observed Markov Decision Process (MAPOMDP) is used to model the dynamic routing problem. Finally, MARL-DR-A2C performs dynamic routing by utilizing current network performance parameters and forecasted congestion parameters by SP-LSTM. The primary objective of MARL-DR-A2C is to train a decentralized routing policy using deep policy gradients. MARL-DR-A2C allows the distributed determination of a shared routing strategy, which solves the dynamic routing problem. In addition, MARL-DR-A2C uses Base Station (BS) associations into its routing decisions so that agents can benefit from information about similar destinations. Therefore, to optimize network efficiency, this MARL-DR-A2C is constantly learning and adjusting the optimum routing policies. Finally, when compared to other recent models on 6G networks, the MARL-DR-A2C outperforms them in simulations.