Spatial Graph-Based Multi-Agent Reinforcement Learning for Asynchronous Coordination with Dynamic Entities

Abstract

Real-world multi-agent systems operate in dynamic environments characterized by asynchrony arising from long-horizon planning, varying numbers of entities, and changes in spatial layouts. Although multi-agent reinforcement learning (MARL) has introduced methods for asynchronous decisionmaking and generalization to variable numbers of agents, unified frameworks that address both challenges simultaneously remain underexplored. In this study, we propose Graph Dynamic MARL (GDM), an actor-critic framework based on a spatial graph representation that is robust to environmental variations. A single gated recurrent unit (GRU)-based actor takes the spatial graph as input and integrates both agent-level and global environmental states, enabling cooperative learning without modifying the model structure. Experimental results in asynchronous environments with dynamic agent configurations and unseen spatial layouts demonstrate that GDM achieves cooperative control under simultaneous asynchrony and structural variation.