Reassessing One-Round Test-Time Refinement for Code Generation

Abstract

Test-time refinement aims to improve generated programs through additional inference, but its value after an initial candidate has been produced remains unclear. We conduct a controlled evaluation of one-round Self-Refine and Self-Debug across seven models and three Python code-generation benchmarks. For each model and task, both methods refine the same initial candidate, allowing us to measure refinement gain without variation in initial generation. Self-Debug produces positive refinement gain in 16 of the 21 model–benchmark combinations and no change in the remaining five, whereas Self-Refine reduces correctness in 16 combinations and improves it in only four. Decomposing refinement gain into repairs and regressions clarifies this contrast. After a diagnostic pass, Self-Debug’s candidate preservation leaves 34.1% of initial final-test failures unaddressed, but 99.95% of initially correct candidates remain correct. With regressions nearly absent, repairs after diagnostic failures translate directly into positive gain. Self-Refine also repairs initial failures, but its regression count is more than three times its repair count overall, producing predominantly negative gain. Resource analysis shows that Self-Refine uses more tokens while generally reducing correctness, whereas Self-Debug provides a more favorable gain–overhead balance, although its inference overhead per net additional pass varies across models and benchmarks. These results show that one-round refinement is not inherently beneficial and should be applied only when its expected gain justifies the additional computation and cost.