Learning with Communication in Multi-agent Deep Reinforcement Learning

Abstract

Communication enables cooperating agents to coordinate their behavior and to access information about the environment. In multi-agent deep reinforcement learning (MADRL), communication protocols can be learned rather than predefined, alongside agents' value functions and policies. However, learning effective communication remains challenging in environments that are stochastic, continuous, or multi-task. This thesis addresses these challenges with the aim of improving learning stability, learning efficiency, and task performance through learnable and adaptive communication. First, we survey the literature on MADRL with communication and propose a classification framework of nine dimensions that capture key aspects of communication design and learning. The framework provides a structured basis for analyzing, comparing, and developing communication methods. Second, we study the Communicating Critics and Decentralized Actors (CCDA) setting, in which critics exchange messages during training while actors act without communication at execution. Under stated assumptions on ideal and noisy communication, we show that the variance of policy gradients under communicating critics is equal to or greater than that obtained with a centralized critic that has access to the complete joint information. To mitigate this variance, we derive a message-dependent baseline, characterize its variance-minimizing form, and propose a complementary regularization technique. Experiments show reduced gradient variance and improved learning performance. Third, we propose Communication with Factorized Policy Gradients (CFPG) for continuous control. CFPG uses centralized but factorized value functions to distinguish the contribution of each communicating agent, and propagates gradients through both policies and message encoders so that communication and action selection are learned jointly. Experiments in continuous-control benchmarks show improved performance and faster learning relative to the evaluated baselines. Finally, we introduce Multi-task Communication Skills (MCS), a Transformer-based method that enables agents to learn and perform multiple tasks with a single shared communication protocol. The protocol accommodates varying numbers of agents as well as heterogeneous observation and action spaces by encoding task-specific observations into a shared message space, limiting unnecessary communication, and aggregating received messages. An additional learning objective encourages messages to carry information about the senders' actions. Together, these contributions provide theoretical insights and practical methods for learning communication in cooperative MADRL. Evaluations on simulated benchmarks demonstrate improvements in learning stability, efficiency, or task performance relative to the evaluated baselines, with the magnitude of improvement varying across tasks.