Skip to content

Improving LLM Collaboration via Multi-Agent Preference Learning

Sep 2026 · arXiv.org · Vol abs/2609.32827 · 0 citations · 77 references
Computer Science

TL;DR

This work instantiates MAPL using MARL from human feedback with a learned reward model and multi-agent direct preference optimization (MADPO), and shows that MAPL can improve collaboration quality and efficiency while approaching the performance of MARL with fixed, well-defined rewards.

Abstract

Several works have explored multi-agent reinforcement learning (MARL) in LLM collaboration. However, constructing reliable rewards is difficult in practice, as complete and accurate metrics are often unavailable and hard to aggregate. Preference learning provides an alternative by learning from comparative human or AI feedback. Yet, its extension to multi-agent systems remains underexplored. To address this gap, we formulate preference-based multi-agent systems (MAS) from decentralized and centralized collaboration perspectives. We also introduce a general multi-agent preference learning framework (MAPL) to solve these problems. MAPL allows iterative updates by comparing the current solution with decentralized or centralized solutions generated by various agents. We instantiate MAPL using MARL from human feedback (MARLHF) with a learned reward model and multi-agent direct preference optimization (MADPO). Experiments on collaborative writing, coding, tool use, and travel planning show that MAPL can improve collaboration quality and efficiency while approaching the performance of MARL with fixed, well-defined rewards. Within MAPL, MARLHF generally outperforms MADPO on most tasks but remains sensitive to data coverage, agent and comparator models, and the underlying MARL algorithms.

View source

Similar papers

Conference Open access Sep 2026

Towards Streamlined Learning and Search for Multi-Agent Optimization

Focusing on multi-agent path finding as an exemplary problem, this paper proposes to simplify two popular approaches to MAPF, namely multi-agent reinforcement learning and adaptive search, to enable seamless combination and transferability of methods without substantial engineering effort.

Thomy Phan · 0 citations

MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimization

This work proposes Multi-Agent Flow-Pretrained Policy Optimization (MA-FPPO), which uses online fine-tuning to improve the cooperative behavior of models pretrained with flow matching through new interactions with the environment.

Guo-Wei Zou, Hao-Nan Chen, Hai-Tao Wang et al. · 0 citations
#large language models Review Open access Sep 2026

A Survey on Reinforcement Learning Optimization Methods for Multi-Agent Collaboration of Large Language Models

The integration of reinforcement learning (RL) into the optimization of multi-agent collaboration for Large Language Models (LLMs) is an important combination of two advanced areas, Multi-Agent Systems (MAS) and LLMs. This paper thoroughly examines the main approaches, evaluation standards, recent progress and existing...

Qian-Ling Zhang · 0 citations

ICMAPE: In-Context Multiagent Pure Exploration

ICMAPE converts the fixed-confidence identification objective into a reward derived from inference confidence, so that standard reinforcement learning machinery can be applied to decentralized pure exploration.

Xin-Yi Hu, Alessio Russo, Aldo Pacchiano · 0 citations
Conference Aug 2026

Curriculum-Aware Advantage Weighting and Coordination Regularization for Multi-Agent RL

Training robust policies in Multi-Agent System (MAS) remains challenging due to high sample complexity, unstable policy updates, and insufficient coordination among agents. Multi-Agent Proximal Policy Optimization (MAPPO) has recently demonstrated strong performance in cooperative multi-agent tasks; however, it does no...

Han-Zhong Zhong, Ying-Jie Jin, Xiang Li · 0 citations
Conference Sep 2026

Role-cognitive decomposition for multi-agent reinforcement learning in StarCraft II micromanagement

In complex, partially observable environments such as real-time strategy games, relying on decentralized multi-agent systems to learn efficient cooperative strategies is a challenging task. This paper proposes a Role Cognition Decomposition algorithm based on Multi-Agent Reinforcement Learning (RCD-MARL), which combine...

Zhen-Chuan Lei, Xian-Zhu Zheng, Zhi-Wei Zhang · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.