This thesis introduces the Multi-Agent LLM (MALLM) framework, which implements and evaluates various decision protocols, namely voting, consensus, and judge decision mechanisms, to simulate multi-agent discussions for conversational task solving and indicates that consensus protocols excel in knowledge-intensive domains while voting and judge protocols are more effective for logic-based tasks.
Abstract
Improving the task performance of Large Language Models (LLMs) is essential, yet scaling these models faces significant challenges such as diminishing returns and high costs. Multi-Agent Systems (MAS) offer a promising solution by distributing tasks among specialized agents to improve the overall task performance. This can reduce training costs at the expense of increased test time due to the discussion and decision-making process. The decision protocol is a critical component of MAS because it specifies how multiple agents collaborate to create a final solution. This thesis introduces the Multi-Agent LLM (MALLM) framework, which implements and evaluates various decision protocols, namely voting, consensus, and judge decision mechanisms, to simulate multi-agent discussions for conversational task solving. Unlike previous work that used a single decision protocol or tested them on limited datasets, this study systematically examines their impact on a diverse set of tasks, ranging from knowledge-based datasets (MMLU, MMLU-Pro, GPQA) and logic-based datasets (StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0). The results indicate that consensus protocols excel in knowledge-intensive domains while voting and judge protocols are more effective for logic-based tasks. Increasing response diversity through independent solution generation improves decision quality, while changes in information access during the decision process have minimal impact.
A novel RAIE taxonomy along four scaling dimensions is proposed, which optimizes the entire thought process through search algorithms and self-verification, and introduces a task-oriented guideline for choosing the best TTS strategy.
Jiayu An, Zheng Chen, Yongcheng Jing et al.· 0 citations
Multi-agent systems powered by large language models (LLMs) have demonstrated potential for collaborative problem-solving, yet increasing the number of agents often introduces redundant reasoning and communication overhead, sometimes degrading performance. We propose AgentDropout, a dynamic strategy inspired by dropout regularization in neural networks, which selectively deactivates low-contribution agents during multi-agent collaboration. At each round of discussion, AgentDropout computes a semantic novelty score for every agent by measuring the divergence of its output relative to the current group consensus. Agents whose novelty score falls below an adaptive threshold are temporarily deactivated, reducing token consumption without sacrificing viewpoint diversity. We evaluate AgentDropout on mathematical reasoning (GSM8K), commonsense reasoning (StrategyQA), and collaborative code generation (HumanEval) tasks. Across three independent runs, AgentDropout achieves accuracy comparable to or modestly above fixed 5-agent debate while reducing total token consumption by 38.0–43.5% and debate rounds by 20.5% on average. Pareto analysis reveals a promising efficiency–quality trade-off, suggesting that dynamic agent deactivation may be useful for deploying multi-agent LLM systems under computational budget constraints.
Zhengxi Xiao, Qi Guo, Yuyue Wang et al.· 2026 8th International Confe...· 1 citation
Multi-agent LLM pipeline systems break down the task among multiple roles for better reasoning, but are benchmarked mainly with large-scale commercial models. In this study, we investigate Parishad, a structured multi-agent system involving five roles, by deploying it on Qwen2.5-7B-Instruct, a local model, on two datasets: GSM8K (500 questions) and HumanEval (164 questions), compared with prompting directly and two-call self-refinement. The multi-agent system drops GSM8K accuracy from 75.0\% to 45.0\% with JSON data format due to the error accumulation problem. With plaintext format, the accuracy is restored to 82.0\%. A two-call self-refinement strategy (V1) can achieve 86.2\% accuracy on GSM8K, with 7.4$\times$ lower token usage. However, the same V1 implementation on HumanEval---where direct accuracy is already 96.3\%---actively destroys performance (66.5\%). A task-aware gated redesign (V2) applied to HumanEval preserves accuracy at 95.1\%. Our results demonstrate that communication format and implementation details determine outcomes more than architectural complexity, and that simpler approaches match or outperform multi-agent pipelines for local 7B model deployment. All code and data are released.
Multi-agent systems built from large language models are deployed widely, yet how much performance is lost when two LLMs must coordinate rather than act alone remains unclear. We formulate the collaboration tax as the team-decentralisation loss of a two-player cooperative game with private information, with two propositions characterising its sign and its equivalence to a max-superadditivity violation. We operationalise this definition on 32 solo-tractable tasks grouped by source of grounding friction and measure it on 11 models from 7 providers. The tax is structured along two no-exception axes: a category ordering across every model and a monotonic decrease with capability. The proximate mechanism is not a reasoning deficit but a four-stage conversational cascade in which agents make ungrounded claims, fail to query the partner, skip integrating both views, and accept the answer without re-derivation. The tax is mechanically predictable from conversation features and partly tractable: a prompt intervention targeting all four stages closes a substantial fraction of the gap, with the dominant bottleneck differing across categories. In heterogeneous pairs the tax is pulled toward the stronger partner rather than the additive midpoint, empirically realising the max-superadditivity violation predicted by our framework. Together these results recast collaboration in LLM systems as a measurable, predictable, and partly tractable cost.
Weixiang Sun, Zehong Wang, Hong Huang et al.· 0 citations
We investigate the use of large language models (LLMs) as evolutionary operators for optimizing the architecture of multi-agent systems (MAS) - a representative instance of LLM-driven optimization over attributed-graph artifacts that arise in generative design. Given a task description, an initial MAS configuration is generated and then refined on a per-task basis through two optimization strategies: (1) a population-based evolutionary approach where LLMs serve as mutation, crossover, and selection operators, with an LLM-as-a-Judge ensemble providing the surrogate fitness signal; and (2) a simpler iterative regeneration scheme that uses textual feedback from LLM judges to guide single-solution improvement without maintaining a population. We evaluate both approaches on the GAIA benchmark. Our experiments reveal a critical sensitivity to the judge model's quality: a capable judge (Claude 4 Sonnet) yields +14.3 and +6.67 percentage-point accuracy improvements for evolutionary and iterative approaches, respectively, whereas a weaker judge (Gemini-2.5-Flash) provides no benefit or even degrades performance below the unoptimized baseline. Both approaches incur substantial computational costs, raising questions about practical cost-effectiveness. These results highlight fundamental challenges of surrogate fitness functions in LLM-driven evolutionary optimization and offer practical insights for applying evolutionary methods to MAS architecture design.
V. Akhmerov, Dmitry Gilemkhanov, Jerzy Kamiński et al.· Proceedings of the Genetic a...· 0 citations
This work introduces the Language Model Council (LMC), a collaborative framework that combines the expertise of multiple specialized AI agents to evaluate a user query from different perspectives and outperforms traditional single-model systems by improving response quality, reducing hallucinations, and increasing user trust through enhanced explainability.
D. M, Shwetha Kr, G. Divya et al.· International Research Journ...· 0 citations