AI Coding Agents on GitHub: Repository Popularity as a Hidden Confounder in the Acceptance of 2.4 Million Agent-Authored Pull Requests

Abstract

Autonomous coding agents now open pull requests on GitHub without a person writing the code, and the share of those pull requests that maintainers merge has become the usual shorthand for how well the agents work. This paper argues that the shorthand measures the projects as much as it measures the agents. Using the public AIDev dataset, a snapshot of 2,743,854 pull requests written by six agents across 326,798 repositories between December 2024 and October 2025, we find that the merge rate falls from 91.58 percent in repositories with no stars to 60.05 percent in repositories with a thousand or more. The decline appears inside every agent separately, and in the five agents with enough volume in the top band to measure it the fall is between 20.3 and 27.6 percentage points, so it cannot be an artefact of which agent is used where. The enriched portion of the dataset that previously published analyses rely on contains only repositories with at least 100 stars, which we verify directly; it holds 2.7 percent of the decided pull requests and merges them at 73.67 percent against 90.63 percent elsewhere. Holding the repository fixed with a Mantel-Haenszel estimator changes the picture between agents: of fifteen comparable pairs, two reverse direction and one loses more than half its effect. Time to a decision separates the same way, a median of 51 seconds in unstarred repositories against 5.4 hours in the most popular ones, with a medium effect size. Even within the enriched subset, 75.8 percent of agent pull requests carry no recorded human review. We conclude that published acceptance figures describe a narrow and unusually demanding slice of GitHub, and we recommend that studies of agent contributions report the popularity distribution of their repositories and estimate effects within repositories rather than across them.