AsynCodeBench is introduced, a dependency-centric benchmark for asynchronous multi-agent software engineering that represents each task with an explicit dependency graph and executable Dependency Checkers and proposes two complementary measures: Asynchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and Dependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution.
Abstract
Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem. Despite the shift from individual problem solving to distributed collaboration, multi-agent systems still lack a direct measure of collaboration and are largely evaluated through task-level outcomes inherited from single-agent coding, conflating individual coding capability with cross-agent coordination. We introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engineering that represents each task with an explicit dependency graph and executable Dependency Checkers. Through this dependency-tracking process, we propose two complementary measures: Asynchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and Dependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution. AsynCodeBench comprises 19 tasks from real-world repositories, exposing 52 directed dependencies as explicit units for evaluating cross-agent collaboration. Experiments across model families, scales, and generations reveal a clear gap between coding and collaboration capability: improvements in coding performance do not necessarily translate into stronger collaboration, and task-level metrics can diverge substantially from dependency-level collaboration measures. Dependency-trajectory analysis further reveals that successful coordination often emerges not gradually, but through concentrated bursts in which many dependencies become resolved over a short portion of the execution trajectory, a pattern we term a hopping window.
MASBench is introduced, a multi-agent collaboration benchmark designed under partially observable constraints that provides deterministic evaluation metrics, including performance score, communication cost, and cost effectiveness, to characterize both collaboration outcomes and communication overhead.
Qi-Zhi Chu, Ze-Kai Yu, Si-Jie Wen et al.· 1 citation
As AI agents increasingly tackle complex repository-level coding tasks, distributing work across multiple agents is a natural way to scale beyond the capabilities of a single agent. To coordinate their interdependent work, these agents share findings and agree on interfaces between modules. However, exchanged informati...
Jia-Qi Xue, Yan-Jun Wang, Xiangci Li et al.· 0 citations
Multi-agent configurations modeled in KERKIS significantly outperform the single-agent baseline on non-trivial scenarios and orchestration-based coordination consumes 47-112% more tokens than alternatives based on standard operating procedures for a comparable quality of results.
Konsta Kalliokoski, François Christophe, Henri Kärkkäinen et al.· Proceedings of the ACM/IEEE...· 0 citations
OpenCollab is introduced, a multi-agent coding framework that provides a unified infrastructure for programmable collaboration and controllable runtime that unifies organization design, enforces experimental control on a shared runtime, and tracks execution through fine-grained event streams.
Chun-Wah Hsu, Kai Gong, Yu Wu et al.· arXiv.org· 3 citations
This work synthesizes a task-conditioned temporal workflow graph that jointly specifies agent connectivity and edge-level communication semantics, and introduces ReActNet, a training-free framework that compiles a query and a set of role-specialized agents into a sequence of directed communication graphs.
Recently, the rapid development of large language models (LLMs) has reshaped software engineering by enabling autonomous code agents that plan, execute, and utilize external tools iteratively to tackle complex tasks. Beyond achieving functional correctness, these agents must faithfully follow process instructions and c...