Mean-Field Critic with Linear Graph Transformer in Sparse-Reward Multi-Agent Reinforcement Learning

Abstract

Scaling cooperative multi-agent reinforcement learning to many agents faces the difficulty of the centralized critic N which has the joint observation space grows with N. Furthermore, it becomes more challenges in the sparse-reward setting due to the sparseness of useful signal. Mean-field approaches address scalability by representing the agent population as a distribution, but sacrifices local graph structure. In this paper, we propose a Linear Graph Transformer approach which replaces the standard centralized critic with a graph-local mean-field critic that aggregates mean-field embeddings only within k-hop graph neighborhoods. The critic preserves local structure by reconstructing the proximity graph from positional observations, so locality is maintained both during rollout and during the critic’s gradient step. The lightweight linear transformer encodes the ordered neighborhood sequence, where reduces the variance from total N agents to a number of neighboring agents in the graph. To evaluate our approach, the experiments are carried on sparse-reward multi-particle environment across different team sizes. The results our proposed method outperforms the baselines in the config with large number of agents substantially.