Beyond the Hype: The Efficiency-Throughput Gap with GitHub Copilot
Abstract
Investigating the real-world impact of GitHub Copilot on software engineering, this field study uncovers a critical disconnect between gains in individual developer efficiency and improvements in organizational throughput. Using a multi-method approach, we employed the Organizational Ohm’s Law framework and integrated engineering productivity surveys, GitHub Copilot and engineering metrics data, and participant information. While we observed positive shifts in engineer motivation and perceived skill enhancement, accompanied by decreased working hours, our analysis revealed no statistically significant improvement in engineering metrics such as monthly pull request volume and lines of code produced. Survey data indicated that engineers found GitHub Copilot most beneficial for generating testing scripts, boilerplate code, queries, and configurations, but less effective for intricate coding tasks and essential human-centric enterprise development activities such as communication and process management. This phenomenon, which we term the efficiency-throughput gap, demonstrates that individual efficiency gains did not translate to greater organizational output. Our findings underscore the complexities of realizing immediate productivity gains with generative AI in software engineering and offer crucial methodological insights and empirical results for designing and evaluating future AI-powered product development. A real-world field study examines how GitHub Copilot affects engineering productivity in an enterprise setting.
