Skip to content

You're Hired: Strategic Model Selection for LLM Collaboration

Sep 2026 · 0 citations · 34 references
Computer Science

TL;DR

It is established that principled and informed team selection is critical and strong model selection algorithms for assembling effective multi-LLM systems are presented.

Abstract

While multi-agent and model collaboration algorithms gain traction to combine the strengths of diverse Large Language Models (LLMs), existing systems remain bottlenecked on pre-defined and hand-crafted model pools. In this work, we investigate the problem of model selection in multi-LLM systems. We propose and systematically evaluate a taxonomy of 9 selection algorithms ranging from diversity of model descriptions, capability-aware behavioral diversity, and LLM-based recruiters. We conduct extensive experiments across two candidate pools of 10 and 32 models, deployed in four model collaboration algorithms, and evaluated across tasks spanning math, coding, QA, and reasoning. Results demonstrate that successful selection algorithms greatly outperform random or heuristics-based teams such as merely selecting the models with top individual performance, by up to 36.1% across settings. Specifically, capability- and training-based selection strategies alleviate selection variance and achieve the best performance, which we recommend to employ before deploying real-world multi-LLM systems. Further analysis reveals that larger candidate pools pose greater challenges to shallow selection heuristics, while algorithms grounded in interacting with candidate models and understanding model capability robustly filter out misaligned, unsafe models, as well as generalizing to novel, out-of-distribution tasks. Together, we establish that principled and informed team selection is critical and present strong model selection algorithms for assembling effective multi-LLM systems.

View source

Similar papers

#artificial intelligence Open access Sep 2026

Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systems

This work systematically evaluates 8 model selection strategies across before-generation (routing) and after-generation (majority-voting, LLM-as-a-judge) MAS architectures on challenging scientific benchmarks and finds that candidate selection within a single model family is the strategy that yields the best relative p...

Sara Vera Marjanovic, Jia-Chen Xu, A. Laptev et al. · 0 citations
#machine learning Preprint Sep 2026

Rethinking the Evaluation of Efficiency Methods for Multi-Agent Systems

This work introduces a controlled and MAS-demanding diagnostic benchmark for representative MAS efficiency methods and shows that many reported gains are setup-dependent and may arise from structural collapse, disabled tool pathways, or starting systems where random pruning already preserves accuracy, rather than robus...

Jia-Mu Zhang, Ling-Xi Zhang, Peng-Jun Lu et al. · 2 citations
Sep 2026

AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

To quantify collaboration effectiveness in addition to conventional binary task success, Causal Collaboration Effectiveness (CCE) is proposed, a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome.

Raphael Shu, Yu-Sen Zhang, Y. Cho et al. · 2 citations
#machine learning Preprint Sep 2026

Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selection

The performance ceiling of an LLM team is constrained not only by individual model capabilities, but also by inter-member error resonance and predictive differences. Although heterogeneous teaming is often observed to be effective in practice, existing approaches lack complementarity metrics that are computable, interp...

Liang-Yu Teng, Heng-Song Liu, Jun-Cen Guo et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

Agent evaluations should be evaluated as configurable systems themselves, and that some desired behaviors are more effectively implemented in the system than requested through prompting, according to a new benchmark of four scientific tasks.

Luis Wiedmann, Leander Girrbach, Cordelia Schmid et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.