RealAIGC: Towards More Realistic Evaluation of AI-Generated Code Detection in Real-World Code Repository

Abstract

AI-generated code (AIGC) from autonomous coding agents are widely adopted across the software development lifecycle. However, AIGC also introduces critical challenges to open-source community governance, making AIGC detection important. Existing AIGC detection benchmarks are fundamentally misaligned with real-world development processes. To address this gap, we construct RealAIGC, a benchmark for AIGC detection grounded in real-world code repositories. RealAIGC is built from the development histories of GitHub and includes pull request (PR) generated by five coding agents alongside human-written code, covering eight major programming languages. It supports multi-granularity detection at the repository, PR, commit, file, and function levels, with more than four million functions. Based on RealAIGC, we conduct a realistic empirical evaluation of 28 AIGC detection methods across three categories. The results show that supervised approaches perform well only in small, homogeneous, single-language repositories, but degrade significantly in mainstream dynamic languages and in large-scale, multi-developer environments. Perplexity based methods are nearly ineffective in realistic settings. No existing method demonstrates general applicability, and all approaches perform unsatisfactorily at the commit and PR levels. This work provides empirical guidance for advancing AIGC detection techniques.