A Dynamic Evaluation Approach to Repository-Level Code Generation Via LLM
Research on LLM-based code generation has achieved strong performance on function-level benchmarks, yet repository-level code generation in real-world projects still faces challenges from cross-file dependencies. Existing studies usually emphasize single-round or final-round outcomes, which limits the analysis of where feedback gains concentrate and when later repair begins to offset earlier progress. We propose a dynamic evaluation framework for repository-level code generation based on DevEval’s Python tasks. The framework constructs processed test sets, evaluates multi-round repair trajectories with unit-test feedback over multiple rounds, and characterizes trajectories via round-level effectiveness, feedback-induced gain, and regression stability. However, roughly three-quarters of the gains emerged within the first three rounds, and net feedback gain turned negative by Round 5. Previously repaired errors reappeared in later iterations, with reappeared error rates from 4.5% to 8.8% across models. These findings indicate that unit-test feedback is most effective as an early-stage signal, and that practical repair systems should adopt early-round budgeting with best-so-far retention rather than open-ended loops.