Focusing on multi-agent path finding as an exemplary problem, this paper proposes to simplify two popular approaches to MAPF, namely multi-agent reinforcement learning and adaptive search, to enable seamless combination and transferability of methods without substantial engineering effort.
Multi-task multi-agent reinforcement learning (MT-MARL) aims to develop a single model capable of solving a diverse set of tasks. However, existing methods often fall short due to the substantial variation in optimal policies across tasks, making it challenging for a single policy model to generalize effectively. In contrast, we find that many tasks exhibit bounded similarity in their underlying dynamics—highly similar within certain groups (e.g., door-open/close) diverge significantly between unrelated tasks (e.g., door-open & object-catch). To leverage this property, we reconsider the role of modularity in multi-task learning, and propose M3W , a novel approach that applies mixture-of-experts (MoE) to world model instead of policy, enabling both learning and planning. For learning, it uses a SoftMoE-based dynamics model alongside a SparseMoE-based predictor to facilitate knowledge reuse across similar tasks while avoiding gradient conflicts across dissimilar tasks. For planning, it evaluates and optimizes actions using the predicted rollouts from the world model, without relying directly on a explicit policy model, thereby overcoming the limitations of policy-centric methods. As the first MoE-based multi-task world model, M3W demonstrates superior performance, sample efficiency, and multi-task adaptability, as validated on Bi-DexHands with 14 tasks and MA-Mujoco with 24 tasks. The code are available at https://github.
Zijie Zhao, Zhongyue Zhao, Kaixuan Xu et al.· Neural Information Processin...· 8 citations
Although Multi-Agent Reinforcement Learning (MARL) has achieved notable success in cooperative tasks, many existing methods struggle to generalize to heterogeneous scenarios, where diversified specialization is required. In such tasks, a common way is to disable parameter sharing among agents, but this alone fails to induce sufficient behavioral specialization. To address this limitation, sequential updating schemes have recently been proposed, where agents are updated one by one based on the behaviors of their predecessors. However, exploration strategies tailored to the sequential setting remain underexplored, leaving untapped potential in utilizing the information flow between agents. In this work, we propose to leverage the sequential access to predecessor information to explicitly enhance both exploration efficiency and policy heterogeneity. We introduce a novel MARL framework, Multi-Agent Divergence Policy Optimization (MADPO) with Mutual Policy Divergence Maximization (Mutual PDM). The proposed Mutual PDM consists of two key components, the intra-agent PDM and the inter-agent PDM. The former one encourages each agent to explore novel behaviors by deviating from its own previous policies across episodes, while the latter one promotes behavioral heterogeneity between the current agent and its predecessors. To improve the stability and directionality of the divergence signals, we employ the Conditional Cauchy-Schwarz (CCS) divergence for Mutual PDM. Further, we propose a new extension of CCS divergence for measuring policy divergence of more than two agents, the Generalized Conditional Cauchy-Schwarz (GCCS) divergence. Our GCCS divergence can guide agents with more informative incentives, and enhance the global heterogenization with multiple policies. We theoretically prove that the proposed two divergences can offer a stable entropy-guided incentive for exploration and heterogenization. We empirically demonstrate the connection between overall performance and policy heterogeneity in heterogeneous MARL. Extensive experiments on three challenging MARL benchmarks with over 25 heterogeneous scenarios demonstrate that MADPO consistently outperforms existing state-of-the-art sequential up dating methods in terms of both overall performance and agent heterogeneity.
Haowen Dou, Lujuan Dang, Mingfei Lu et al.· IEEE Transactions on Pattern...· 0 citations
Despite recent advances in multi-agent path finding, achieving robust coordination in dynamic and crowded warehouse environments remains a bottleneck due to training instability and inefficient credit assignment. To address these challenges, we propose BeMapper, a novel evolutionary-augmented reinforcement learning framework that integrates a multi-agent bidirectionally-coordinated network (BicNet) with a distributed actor-critic architecture. Technically, our core novelty lies in three aspects: (1) A bidirectional feature fusion mechanism that enables agents to perceive collective spatial states beyond local observations; (2) An evolutionary-driven critic selection strategy that iteratively propagates high-performing models to accelerate convergence; (3) A multi-metric scoring system that incorporates success rate variance to penalize unstable behaviors and resolve credit assignment ambiguity. Extensive experiments demonstrate the superiority of BeMapper: it achieves a 98.66% mean success rate, outperforming state-of-the-art baselines Mapper (95.51%) and BicNet (93.78%) by 3.15%and 4.88%, respectively. Crucially, BeMapper yields a significantly higher average reward of 18.81, representing a relative improvement of 1.65 over Mapper and a substantial leap over BicNet's near-zero performance (0.04). Furthermore, in more crowded scenarios, BeMapper reduces the average travel steps to 36, being 5-9 steps shorter than competing methods, effectively enhancing operational throughput while ensuring robustness for large-scale industrial automation.
One of the main challenges in the field of artificial intelligence, and reinforcement learning (RL) in particular, is the development of generalist and flexible agents capable of solving multiple tasks—each requiring the agent to learn a potentially new, specialized behavior. Tackling this challenge requires agents to learn behaviors that may involve optimizing a single objective, or trading off between multiple conflicting objectives. In this thesis, we study how to design flexible RL agents that can, in a sample-efficient manner, adapt their behavior to solve any given tasks—each of which is defined by multiple (possibly conflicting) objectives. We introduce new multi-policy methods that empower RL agents to (i) carefully learn multiple behaviors, each specialized in a particular task; and (ii) combine previously-learned behaviors to efficiently identify solutions to novel tasks, which, importantly, may require the agent to assign different preferences to each of its new objectives. The methods we introduce have strong theoretical guarantees regarding the optimality of the set of behaviors learned by agents and their capability to solve new tasks in a zero-shot manner, even in the presence of function approximation errors. We evaluate the proposed methods in various challenging multi-task and multi-objective RL problems and show that our algorithms outperform various current state-of-the-art methods in domains with both discrete and continuous state and action spaces.
L. N. Alegre, Ana L. C. Bazzan, Bruno C. da Silva· Anais do XXXIX Concurso de T...· 0 citations
Multi-agent systems in the real-world (e.g., drone swarms, autonomous cars, warehouse robots) must satisfy rich, temporal tasks while avoiding collisions. Signal Temporal Logic (STL) elegantly encodes such objectives, but current STL planning methods face critical limitations. State-of-the-art optimization-based approaches can handle arbitrary STL specifications but struggle with scalability, becoming computationally impractical as the number of agents grows. Learning-based methods efficiently handle a large number of agents with rapid planning times but fare poorly when deployment-time objectives differ from those used during training, and do not support planning tasks that require different specifications to be ascribed to different agents (i.e., heterogeneity) or team-level specifications requiring coordination of multiple agents. This fundamental trade-off between generalizability and scalability presents a challenge for realizing multi-agent STL planning algorithms in practice. To overcome this challenge, we introduce a new diffusion method for multi-agent planning with STL specifications. Using a differentiable approximation of STL, we integrate the STL gradient in the denoising process, making our approach generalizable to novel formulas whose predicates are placed anywhere within the goal region covered during training, while achieving the same scalability as existing learning-based methods. Our method supports heterogeneous specifications, and by using diffusion models, naturally enhances plan diversity, thereby significantly reducing safety-related violations (e.g., collisions) among agents. A detailed evaluation study justifies the utility of STL-guided diffusion-based multi-agent planners for constructing generalizable, scalable, and diverse plans. Videos and code are available at https://www.jeappen.com/diff-ma-stl/ and https://github.com/jeappen/diff-ma-stl .
Joe Eappen, Zikang Xiong, S. Iyengar et al.· 0 citations
This work presents OptiMAS, a task-agnostic agentic optimizer that leverages textual interaction trajectories and task feedback as loss signals for end-to-end MAS evolution and sustains performance improvement over extended optimization horizons.
Yuxin Cheng, Chang Liu, Hanxin Yu et al.· 0 citations
