The Role of Network Topology and Opponent Information in Shaping Cooperation in Multi-Agent Reinforcement Learning Systems
TL;DR
Experimental results show that the number of neighbours per node and the average path length are the main factors affecting the emergence of cooperation in scenarios in which each agent learns to play the two-player Iterated Prisoner's Dilemma using deep reinforcement learning.
Abstract
Several works have investigated the influence of graph topology on cooperation among artificial agents, while the majority of the literature has focused on modelling agents'adaptation through strategy imitation, which relies solely on the cumulative payoffs of others. This paper investigates scenarios in which each agent learns to play the two-player Iterated Prisoner's Dilemma (IPD) using deep reinforcement learning. Each agent is represented as a node in a graph, where its neighbours constitute the pool of opponents with whom it can interact. During each IPD episode, agents are provided with different types of information about their opponent, consisting of action history and opponent identity. Experimental results across different graph topologies show that the number of neighbours per node and the average path length are the main factors affecting the emergence of cooperation. We also show that, while partner selection fosters mutual cooperation by limiting the diversity of the opponent pool, providing agents with the identity of their opponent hinders the proliferation of cooperative strategies.
