It is established that principled and informed team selection is critical and strong model selection algorithms for assembling effective multi-LLM systems are presented.
Abstract
While multi-agent and model collaboration algorithms gain traction to combine the strengths of diverse Large Language Models (LLMs), existing systems remain bottlenecked on pre-defined and hand-crafted model pools. In this work, we investigate the problem of model selection in multi-LLM systems. We propose and systematically evaluate a taxonomy of 9 selection algorithms ranging from diversity of model descriptions, capability-aware behavioral diversity, and LLM-based recruiters. We conduct extensive experiments across two candidate pools of 10 and 32 models, deployed in four model collaboration algorithms, and evaluated across tasks spanning math, coding, QA, and reasoning. Results demonstrate that successful selection algorithms greatly outperform random or heuristics-based teams such as merely selecting the models with top individual performance, by up to 36.1% across settings. Specifically, capability- and training-based selection strategies alleviate selection variance and achieve the best performance, which we recommend to employ before deploying real-world multi-LLM systems. Further analysis reveals that larger candidate pools pose greater challenges to shallow selection heuristics, while algorithms grounded in interacting with candidate models and understanding model capability robustly filter out misaligned, unsafe models, as well as generalizing to novel, out-of-distribution tasks. Together, we establish that principled and informed team selection is critical and present strong model selection algorithms for assembling effective multi-LLM systems.
This work systematically evaluates 8 model selection strategies across before-generation (routing) and after-generation (majority-voting, LLM-as-a-judge) MAS architectures on challenging scientific benchmarks and finds that candidate selection within a single model family is the strategy that yields the best relative p...
Sara Vera Marjanovic, Jia-Chen Xu, A. Laptev et al.· Proceedings of the Second Wo...· 0 citations
This work introduces a controlled and MAS-demanding diagnostic benchmark for representative MAS efficiency methods and shows that many reported gains are setup-dependent and may arise from structural collapse, disabled tool pathways, or starting systems where random pruning already preserves accuracy, rather than robus...
Jia-Mu Zhang, Ling-Xi Zhang, Peng-Jun Lu et al.· 2 citations
To quantify collaboration effectiveness in addition to conventional binary task success, Causal Collaboration Effectiveness (CCE) is proposed, a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome.
Raphael Shu, Yu-Sen Zhang, Y. Cho et al.· arXiv.org· 2 citations
The performance ceiling of an LLM team is constrained not only by individual model capabilities, but also by inter-member error resonance and predictive differences. Although heterogeneous teaming is often observed to be effective in practice, existing approaches lack complementarity metrics that are computable, interp...
Liang-Yu Teng, Heng-Song Liu, Jun-Cen Guo et al.· 0 citations
Agent evaluations should be evaluated as configurable systems themselves, and that some desired behaviors are more effectively implemented in the system than requested through prompting, according to a new benchmark of four scientific tasks.
Luis Wiedmann, Leander Girrbach, Cordelia Schmid et al.· 0 citations
This work introduces Steiner's taxonomy of group tasks as a framework for analyzing multi-agent LLM scaling and shows that task structure is a fundamental determinant of team scaling.