Aug 2026· IEEE Transactions on Automation Science and Engineering· Vol 23, pp. 14725-14741· 0 citations· 58 references
Computer Science
TL;DR
This paper introduces Multi-AGent Preference-Integrated lEarning (MAGPIE), a framework that leverages agent-specific preference signals in the multi-agent learning process and can derive Nash equilibrium solutions.
Abstract
Multi-agent reinforcement learning (MARL) is a powerful framework for solving complex collaborative tasks, but it relies heavily on well-defined global reward functions. Designing such rewards is challenging, especially in systems with heterogeneous agents, where a single scalar objective may fail to capture diverse behaviors. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), which addresses these challenges through agent-specific preference modeling. Each agent is evaluated by a dedicated expert through preference signals, eliminating the need for global evaluation. We theoretically prove that optimizing these decentralized pReferences converges to a Nash equilibrium policy. To integrate local preferences into a coherent global objective, we construct agent-specific reward models from preference data and combine them via a monotonic aggregation mechanism. We further prove that optimizing this aggregate reward model is equivalent to training the Nash equilibrium policy. Extensive experiments on benchmark multi-agent tasks and a sequential production line task show that MAGPIE achieves performance comparable to reward-engineered baselines, demonstrating its potential to facilitate policy learning in scenarios where precise reward engineering is impractical. Note to Practitioners—Multi-agent systems are widely used in modern engineering applications. For example, autonomous vehicle fleets coordinate to prevent collisions while maintaining efficiency, and industrial manufacturing lines work together to meet production targets without causing buffer overflows. Multi-agent reinforcement learning (MARL) provides a powerful framework for enabling such collaboration, but its success depends heavily on well-designed reward functions. Designing these rewards is often challenging, especially when agents play distinct roles, as it is difficult to translate complex interactions and diverse agent behaviors into precise numerical signals. In contrast, providing comparative feedback on preferred behaviors is often more intuitive than specifying explicit mathematical rewards. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), a framework that leverages agent-specific preference signals in the multi-agent learning process. MAGPIE learns agent-specific reward models and combines them into a unified global objective using monotonic aggregation. By optimizing this objective, we can derive Nash equilibrium solutions. Importantly, preferences can be provided by lightweight automated rules or domain-specific heuristics, eliminating the need for costly human annotators. MAGPIE is effective, easy to implement, and particularly suitable for complex systems where traditional reward design is impractical.
One of the main challenges in the field of artificial intelligence, and reinforcement learning (RL) in particular, is the development of generalist and flexible agents capable of solving multiple tasks—each requiring the agent to learn a potentially new, specialized behavior. Tackling this challenge requires agents to learn behaviors that may involve optimizing a single objective, or trading off between multiple conflicting objectives. In this thesis, we study how to design flexible RL agents that can, in a sample-efficient manner, adapt their behavior to solve any given tasks—each of which is defined by multiple (possibly conflicting) objectives. We introduce new multi-policy methods that empower RL agents to (i) carefully learn multiple behaviors, each specialized in a particular task; and (ii) combine previously-learned behaviors to efficiently identify solutions to novel tasks, which, importantly, may require the agent to assign different preferences to each of its new objectives. The methods we introduce have strong theoretical guarantees regarding the optimality of the set of behaviors learned by agents and their capability to solve new tasks in a zero-shot manner, even in the presence of function approximation errors. We evaluate the proposed methods in various challenging multi-task and multi-objective RL problems and show that our algorithms outperform various current state-of-the-art methods in domains with both discrete and continuous state and action spaces.
L. N. Alegre, Ana L. C. Bazzan, Bruno C. da Silva· Anais do XXXIX Concurso de T...· 0 citations
This paper develops a multi-agent reinforcement learning-based (MARL) delegation training that enables agents to make sequential delegation decisions while minimizing the total execution cost and introduces two new frameworks for collaboration and delegation in multi-agent systems.
Ziqing Lu, Avinash Mudireddy, Sarra M. Alqahtani et al.· 0 citations
Reinforcement learning (RL) provides a framework for sequential decision making under explicit objectives. In its classical form, RL studies how an agent should act to maximise long-term reward in a dynamic environment. In richer settings, the problem extends beyond a single agent and fixed environment: intelligent behavior may require strategic interaction, adaptation to uncertainty, and reasoning over high-dimensional worlds. This thesis studies RL from two perspectives: algorithms in games and RL in the era of foundation models. The first part focuses on multi-agent RL in games. It examines how incentives, policies, and equilibrium concepts interact in competitive and general-sum environments, spanning two-player zero-sum games, large-scale video games, and multi-player settings with general structure. These works investigate learning in multi-agent systems and the behavior of RL methods in interactive environments. The second part studies RL with generative and foundation models, motivated by the idea that prior knowledge can enrich sequential decision making. Pretrained generative models and learned world models serve as representation tools and structured priors for planning, control, and policy optimization. The thesis develops diffusion-based world models, investigates RL for efficient video generation, explores generative models as policy classes, and studies interactive video world models in which actions shape future observations. It also addresses long-horizon modeling through architectures with memory. Together, these contributions present a unified view of RL as objective-driven adaptation in complex sequential domains. From strategic games to generative world models, the thesis highlights how RL connects decision making, environment modeling, and emerging foundation-model capabilities, offering a broader perspective on the principles underlying intelligent behavior.
Although Multi-Agent Reinforcement Learning (MARL) has achieved notable success in cooperative tasks, many existing methods struggle to generalize to heterogeneous scenarios, where diversified specialization is required. In such tasks, a common way is to disable parameter sharing among agents, but this alone fails to induce sufficient behavioral specialization. To address this limitation, sequential updating schemes have recently been proposed, where agents are updated one by one based on the behaviors of their predecessors. However, exploration strategies tailored to the sequential setting remain underexplored, leaving untapped potential in utilizing the information flow between agents. In this work, we propose to leverage the sequential access to predecessor information to explicitly enhance both exploration efficiency and policy heterogeneity. We introduce a novel MARL framework, Multi-Agent Divergence Policy Optimization (MADPO) with Mutual Policy Divergence Maximization (Mutual PDM). The proposed Mutual PDM consists of two key components, the intra-agent PDM and the inter-agent PDM. The former one encourages each agent to explore novel behaviors by deviating from its own previous policies across episodes, while the latter one promotes behavioral heterogeneity between the current agent and its predecessors. To improve the stability and directionality of the divergence signals, we employ the Conditional Cauchy-Schwarz (CCS) divergence for Mutual PDM. Further, we propose a new extension of CCS divergence for measuring policy divergence of more than two agents, the Generalized Conditional Cauchy-Schwarz (GCCS) divergence. Our GCCS divergence can guide agents with more informative incentives, and enhance the global heterogenization with multiple policies. We theoretically prove that the proposed two divergences can offer a stable entropy-guided incentive for exploration and heterogenization. We empirically demonstrate the connection between overall performance and policy heterogeneity in heterogeneous MARL. Extensive experiments on three challenging MARL benchmarks with over 25 heterogeneous scenarios demonstrate that MADPO consistently outperforms existing state-of-the-art sequential up dating methods in terms of both overall performance and agent heterogeneity.
Haowen Dou, Lujuan Dang, Mingfei Lu et al.· IEEE Transactions on Pattern...· 0 citations
Quality-Diversity (QD) methods combined with policy gradients have shown strong performance in single-agent reinforcement learning, but extending them to multi-agent settings introduces challenges from partial observability and agent interactions. We propose MAPGA-ME, a multi-agent extension of PGA-MAP-Elites that integrates policy gradient updates into MAP-Elites for cooperative control. Our results show that directly transferring policy gradient mechanisms from single-agent QD does not consistently improve performance in multi-agent environments. In particular, a design choice effective in single-agent settings becomes less suitable under decentralized, partially observable conditions. Across multiple configurations, we identify key factors affecting the effectiveness of policy gradient-based QD in multi-agent learning, providing practical guidance for adapting these methods.
Hai D. Pham, Ngoc Hoang Luong· Proceedings of the Genetic a...· 0 citations
Multi-task multi-agent reinforcement learning (MT-MARL) aims to develop a single model capable of solving a diverse set of tasks. However, existing methods often fall short due to the substantial variation in optimal policies across tasks, making it challenging for a single policy model to generalize effectively. In contrast, we find that many tasks exhibit bounded similarity in their underlying dynamics—highly similar within certain groups (e.g., door-open/close) diverge significantly between unrelated tasks (e.g., door-open & object-catch). To leverage this property, we reconsider the role of modularity in multi-task learning, and propose M3W , a novel approach that applies mixture-of-experts (MoE) to world model instead of policy, enabling both learning and planning. For learning, it uses a SoftMoE-based dynamics model alongside a SparseMoE-based predictor to facilitate knowledge reuse across similar tasks while avoiding gradient conflicts across dissimilar tasks. For planning, it evaluates and optimizes actions using the predicted rollouts from the world model, without relying directly on a explicit policy model, thereby overcoming the limitations of policy-centric methods. As the first MoE-based multi-task world model, M3W demonstrates superior performance, sample efficiency, and multi-task adaptability, as validated on Bi-DexHands with 14 tasks and MA-Mujoco with 24 tasks. The code are available at https://github.
Zijie Zhao, Zhongyue Zhao, Kaixuan Xu et al.· Neural Information Processin...· 8 citations