Jul 2026· CCF Transactions on Pervasive Computing and Interaction· Vol 8, pp. 483 - 499· 0 citations· 42 references
TL;DR
This paper proposes effective multi-agent selective learning methods to boost sample-efficient training by learning from successful experiences, and adopts a retrogression-based selection method to identify successful agent trajectories from the team rewards.
This paper introduces Multi-AGent Preference-Integrated lEarning (MAGPIE), a framework that leverages agent-specific preference signals in the multi-agent learning process and can derive Nash equilibrium solutions.
Ni Mu, Yao Luan, Yiqin Yang et al.· IEEE Transactions on Automat...· 0 citations
Although Multi-Agent Reinforcement Learning (MARL) has achieved notable success in cooperative tasks, many existing methods struggle to generalize to heterogeneous scenarios, where diversified specialization is required. In such tasks, a common way is to disable parameter sharing among agents, but this alone fails to induce sufficient behavioral specialization. To address this limitation, sequential updating schemes have recently been proposed, where agents are updated one by one based on the behaviors of their predecessors. However, exploration strategies tailored to the sequential setting remain underexplored, leaving untapped potential in utilizing the information flow between agents. In this work, we propose to leverage the sequential access to predecessor information to explicitly enhance both exploration efficiency and policy heterogeneity. We introduce a novel MARL framework, Multi-Agent Divergence Policy Optimization (MADPO) with Mutual Policy Divergence Maximization (Mutual PDM). The proposed Mutual PDM consists of two key components, the intra-agent PDM and the inter-agent PDM. The former one encourages each agent to explore novel behaviors by deviating from its own previous policies across episodes, while the latter one promotes behavioral heterogeneity between the current agent and its predecessors. To improve the stability and directionality of the divergence signals, we employ the Conditional Cauchy-Schwarz (CCS) divergence for Mutual PDM. Further, we propose a new extension of CCS divergence for measuring policy divergence of more than two agents, the Generalized Conditional Cauchy-Schwarz (GCCS) divergence. Our GCCS divergence can guide agents with more informative incentives, and enhance the global heterogenization with multiple policies. We theoretically prove that the proposed two divergences can offer a stable entropy-guided incentive for exploration and heterogenization. We empirically demonstrate the connection between overall performance and policy heterogeneity in heterogeneous MARL. Extensive experiments on three challenging MARL benchmarks with over 25 heterogeneous scenarios demonstrate that MADPO consistently outperforms existing state-of-the-art sequential up dating methods in terms of both overall performance and agent heterogeneity.
Haowen Dou, Lujuan Dang, Mingfei Lu et al.· IEEE Transactions on Pattern...· 0 citations
Multi-task multi-agent reinforcement learning (MT-MARL) aims to develop a single model capable of solving a diverse set of tasks. However, existing methods often fall short due to the substantial variation in optimal policies across tasks, making it challenging for a single policy model to generalize effectively. In contrast, we find that many tasks exhibit bounded similarity in their underlying dynamics—highly similar within certain groups (e.g., door-open/close) diverge significantly between unrelated tasks (e.g., door-open & object-catch). To leverage this property, we reconsider the role of modularity in multi-task learning, and propose M3W , a novel approach that applies mixture-of-experts (MoE) to world model instead of policy, enabling both learning and planning. For learning, it uses a SoftMoE-based dynamics model alongside a SparseMoE-based predictor to facilitate knowledge reuse across similar tasks while avoiding gradient conflicts across dissimilar tasks. For planning, it evaluates and optimizes actions using the predicted rollouts from the world model, without relying directly on a explicit policy model, thereby overcoming the limitations of policy-centric methods. As the first MoE-based multi-task world model, M3W demonstrates superior performance, sample efficiency, and multi-task adaptability, as validated on Bi-DexHands with 14 tasks and MA-Mujoco with 24 tasks. The code are available at https://github.
Zijie Zhao, Zhongyue Zhao, Kaixuan Xu et al.· Neural Information Processin...· 8 citations
Multi-agent Reinforcement learning has gained significant attention for solving decision-making problems involving multiple autonomous agents. However, effective learning in MARL is still difficult due to environments, dependencies between agents, and poor exploration strategies. Although adaptive exploration and curriculum learning methods, such as Reward Prediction Error Adaptive Learning (RPEAL) along with Reward-Shaped Adaptive Curriculum Learning (RSACL), have produced good outcomes in single-agent reinforcement learning, their use in multi-agent contexts has not been thoroughly investigated. In this research, RPEAL and RSACL are introduced. This paper extends the previous single-agent work to the broader realm of cooperative multi-agent reinforcement learning. The introduced adaptive control mechanism are integrated into several popular multi-agent algorithms such as IPPO, CPPO, MADDPG, and MASAC are empirically compared in a standard petting-zoo environments. The experimental evaluation shows gains in these algorithms upon the introduction of adaptive control mechanisms, where the centralized critic outperforms the individual learners in terms of stability and convergence. Unlike previous works, which only considered single-agent reinforcement learning, in this paper we extend the RPEAL and RSACL to the multi-agent domain. To be specific, we propose team reward prediction error modeling with a centralized critic, as well as performance-driven curriculum learning for multi-agents.
B. Adwaith, Kevin Francis, Remya Nair T· International Conference on...· 0 citations
Real-world agent learning is often constrained by costly environment interactions, such as running time-consuming experiments or obtaining human feedback. In-context learning offers a highly sample-efficient way for agents to learn from their own interaction histories, but its gains disappear once that experience is removed from the context. Separately, context distillation provides a mechanism for internalizing contextual information into model weights. However, applying it to agents'interaction histories without sacrificing environment sample efficiency remains underexplored. We term this problem Experience Distillation and develop an implementation that requires no further environment interaction beyond the collected experience. Experiments on 749 curated software-engineering tasks and six text-adventure games show that it retains at least 64.8\% of the gains from in-context learning across both domains, whereas direct supervised fine-tuning on the collected experience recovers only 3.8\%. Compared with classical reinforcement-learning baselines, in-context learning from trial-and-error experience followed by Experience Distillation matches their performance with at least \(9.6\times\) fewer environment samples.
Chenhui Gou, Haoqin Tu, Yunhao Fang et al.· 0 citations
As autonomous agents are increasingly deployed across diverse operational contexts, aligning their behavior with human intent demands reward functions that remain robust to such changes rather than overfitting to any single environment. Inverse reinforcement learning (IRL) provides a principled way to infer such objectives from human feedback. However, existing analyses of optimal teaching approaches for IRL focus on single-environment, demonstration-only settings, leaving underexplored how heterogeneous feedback modalities and environment dynamics jointly constrain reward functions that generalize across multiple environments. Because demonstrations in one MDP entangle reward information with that environments specific structure, the resulting rewards frequently fail to generalize when the agent is deployed in a new setting. We first analyze how different feedback modalities constrain rewards, showing that, in the unlimited-data regime, comparisons impose strictly stronger global constraints than other modalities. Beyond this theoretical analysis, we introduce a hierarchical machine teaching algorithm for reward learning that operates across multiple MDPs. The algorithm first greedily selects informative environments that expose complementary reward constraints, then strategically queries low-cost feedback within those environments. Empirically, our method achieves substantially lower regret and stronger generalization to held-out environments than uniform teaching baselines under identical feedback budgets, demonstrating the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions.
Ali Larian, Qian Lin, Changzhong Wu et al.· 0 citations