Jul 2026· Annual International Computer Software and Applications Conference· pp. 2126-2131· 0 citations· 20 references
Abstract
Autonomous AI agents are increasingly being used to assist software developers in tasks such as code generation and refactoring. However, we still lack a clear understanding of how these agents interact with humans in collaborative workflows such as GitHub pull requests (PRs). A helping hand to study this phenomenon comes from the AIDev dataset featuring 932k PRs authored (i.e., opened) by AI agents across 116k GitHub repositories. Nevertheless, such a dataset provides a limited view of the collaborations between humans and agents, since the latter may also be involved in PRs opened by humans. In this work, we analyse interaction patterns between humans and AI agents in PRs. To support this analysis, we introduce a novel methodology that extends the AIDev dataset to capture all PRs involving agent participation, irrespective of whether the agents initiated the PR. We apply such a methodology to a subset of 383 repositories from AIDev, finding 6,445 PRs, out of which 5,118 are agentrelevant (vs the 3,197 of AIDev on the same repositories) for a total of 74,525 interactions from 671 unique users. We classify these PRs by authorship (i.e., agent, bot, or human) and examine which participant types perform key interactions (e.g., reviews). Our findings indicate that human-agent collaboration is the most common interaction pattern, with about nine out of ten changes being merged. Although agents author a substantial proportion of PRs, fully autonomous contributions remain uncommon and are rarely accepted. These results highlight the increasing importance of AI agents in augmenting human capabilities in software development, as well as the central role of human oversight in ensuring successful contributions.
AI coding agents may generate and submit Pull Requests (PRs) to the same repository at the same time. However, research concerning the extent of concurrent submission by AI coding agents to a common repository does not exist. This paper uses the AIDev-pop dataset (33,596 PRs in 2,807 repositories) to provide the first empirical examination of the prevalence of concurrent submission using PRs authored by agents. We report that when considering exact temporal overlap, 40.2% of repositories contain co-active agent-authored PR pairs; further, the co-active pairs account for 79.4% of all PRs generated by an AI agent. When we examine co-activity within a one week collaboration window, the percentages are increased to 53.4% and 95.0%, respectively. For the majority of the co-active PR pairs (underlying the vast majority of which are intra-agent authored), both PRs were authored by the same agent, while only 0.5% of co-active pairs were cross-agent, and occurred in only 122 out of 2807 total repositories examined (or approximately 4.3%). Additionally, we replayed actual three way git merges on 747 unique co-active pairs (one per repository), and computed the percentage of textual conflict encountered during the merge operation to combine the two PRs in each pair. We observed that the percentage of textual conflict encountered was significantly higher for cross-agent pairs compared to intra-agent pairs: 41.7% vs. 19.8%, respectively, with non-overlapping 95% confidence intervals. Lastly, we developed a classification system based on the detection of conflict reported by git, and determined that the majority of conflicts resulted from modifications to source code files (84.4% of conflicted files) and not dependency manifest files; further, nearly 42% of conflicts we observed were structural (i.e., modify/delete or add/add).
George Xu, Arjun Subramanian, N. Karthik· 0 citations
Large-scale human–agent collaboration is investigated by comparing 40,214 pull requests across 2,807 GitHub repositories, including 33,596 agent-authored PRs from five autonomous coding agents and 6,618 human-authored PRs, revealing a socio-technical trade-off.
A. Njoku, Zohreh Sharafi, F. Khomh· AIware· 2 citations
For our Special Interest Group (S.I.G.), we propose papers that address the orchestration of teams by synthesizing their workflows into a coherent whole, whether these teams are composed of human, machine, Generative AI (gen-AI), robot or AI-Agentic members. The bigger picture of interdependence, teamwork and Gen-AI indicates the need by organizations to build a library of human and artificial agents with bidirectional agency (responsibility) to achieve operational goals (missions), considering agentic risk tolerances, available skills, and vulnerabilities across a complex trade space among the skills available versus those needed for the tasks assigned to complete an operation. In this trade space, agents (human or artificial) from multiple systems with the requisite skills to accomplish a designated task and timeline combined to form a hierarchy of humans, robots, machines and AI. This complex system produces workflows that must be synthesized into a unit(s), then orchestrated to accomplish the goals assigned to it, yet remain trusted even in competitive and uncertain environments. Once synthesized into a unit (e.g., a team), Gen-AI provides the opportunity to not only advance the science of teams by orchestrating team products and performances, but also has raised several concerns (viz., AI used for deception, superintelligence, blackmail, or existential threats to humans). For our S.I.G., We are interested in orchestrating teams: What are the benefits, drawbacks, and, most importantly, can humans, machines and Agentic AI be synthesized and managed (orchestrated)?
W. Lawless, Marco Brambilla, Stephen Russell· AHFE International· 0 citations
AI coding agents increasingly submit complete pull requests (PRs) with minimal human intervention, shifting software development from AI-assisted to autonomous workflows. As these agents become more prevalent, ensuring the code they generate is adequately tested, by existing tests or by tests the agents write, is critical to preventing regressions, yet little is known about testing in agentic PRs. To address this gap, we analyze 4882 agent-generated PRs from the AIDev dataset (532 Java and 4350 Python PRs) produced by five coding agents. We study (i) how often agents include test changes and (ii) how well covered are code changes by existing and agent-written tests. Agents include test changes in only 49.6% of PRs that change code under test files. Existing tests provide an incomplete safety net: they cover 61.5% of agents'changed executable lines in Java and only 27.0% in Python, where 64.8% of PRs have no changed line executed by any existing test. Agent-written tests improve coverage over existing tests, but only in a minority of PRs: 35.9% of Java and 22.5% of Python Code + Tests PRs show a coverage gain. Across both languages, error-handling constructs (e.g., try and catch blocks) are the most consistently under-tested, with miss rates reaching 86.0% in Java and 81.0% in Python. These findings motivate coverage-aware development practices, coverage feedback loops for coding agents, and evaluation benchmarks that measure test quality to better help agents reliably test their own code.
A. Dipongkor, Talank Baral, Wing Lam et al.· 0 citations
AI coding agents are increasingly integrated into software development workflows, operating on both sides of the pull-request (PR) process: AI authoring agents create or modify PRs, while AI reviewers evaluate them. This creates a closed loop in which one AI coding agent reviews a contribution attributed to another. We construct a large-scale dataset of AI-to-AI code review by linking AI-attributed PRs with AI-attributed review events from CodAGE, a public dataset of coding-agent-generated GitHub events. Our dataset contains 248,641 unique AI-attributed PRs that received at least one AI-attributed review. Of these, 45,269 received cross-product review and 208,145 received same-product review; 4,773 PRs received both. Cross-product AI-to-AI review occurred in approximately 1.6% of identified agent-authored PRs but was substantial in absolute terms, and its volume increased by more than two orders of magnitude from 2025-Q1 to 2025-Q3. Reviewer output varied across author-reviewer configurations. CodeRabbit labeled 35.0% of its comments on Claude Code-authored PRs as refactor comments, compared with 10.5% on Copilot-authored PRs, although this difference may reflect characteristics of the PRs rather than the reviewer. For three of four dual-role reviewers, mean comments per PR were 58-65% higher in the same-product group, although effect sizes were small or negligible and the difference was concentrated in the upper tail. Among pairs with complete, nonnegative timestamps, the observed median latency was 1.2 minutes for cross-product pairs and 4.7 minutes for same-product pairs; differential timestamp availability and reviewer composition limit this comparison. Overall, closed-loop AI-to-AI review is increasing but remains a minority of identified agent activity, with review output varying across authoring-agent groups and product configurations.
AI agents are becoming a fundamental part of modern software creation, helping developers in generating code, debugging, designing systems, etc. But there is a clear difference between how beginners and experienced software engineers get benefits from these tools. Newbies usually depend on agents for one-time prompts and quick answers, whereas mature users utilize them through well-defined, repeated workflows that raise productivity and consistency. In this article, we discuss this difference and emphasize that getting the full potential does not merely depend on better prompts but on workflows driven by instructions developers create clear and reusable instruction files to direct agent behavior across tasks. When developers stop seeing agents only as chat interfaces but as programmable collaborators, they can produce more reliable and high-quality outputs. We offer in our paper methods like designing modular instructions, narrowing down the context, and iterative refinement loops, as well as a case study illustrating how a team made a code review more efficient and minimized the rework by making agent instructions standard. The results stress that structured forms of interaction rather than sporadic use are the main ways to tap into advanced features. Our paper provides a conceptual model for agent usage at large scale, hands-on advice for the implementation of instruction files in actual settings, and validation that skillful developers can far exceed basic usage by adopting orderly, system-like approaches to agent collaboration.
Madhurima Kommuru, Srujana Pulipaka· International Journal of Mod...· 0 citations