Skip to content
Log in
Preprint

RLPF: Reinforcement Learning from Performance Feedback for Code Generation

Huihao Jing Haozhe Cui Wenbin Hu Shaojin Chen Haochen Shi Changxuan Fan Yuxuan Liu Hanyu Yang Sirui Zhang Ziyi Chen Haoran Li Yangqiu Song
Jul 2026 · 0 citations · 37 references
Computer Science

TL;DR

This work proposes RLPF, reinforcement learning from performance feedback, which turns execution outcomes into a staged reward, and suggests that code agents can be trained not only to pass tests, but also to optimize the programs they write.

Abstract

Code models are increasingly trained with execution feedback, but most training signals still stop at correctness. This leaves an important gap for systems code: two programs can pass the same tests while differing greatly in runtime. We study how to train code agents to prefer faster correct implementations, rather than treating efficiency only as an evaluation metric. The key difficulty is that runtime is a fragile reward. It is meaningful only after a program is correct, varies across tasks, and gives little guidance when most sampled programs fail to compile or run. We propose \textbf{RLPF}, reinforcement learning from performance feedback, which turns execution outcomes into a staged reward. Failed programs are ordered by execution progress, while correct programs are ranked by their relative improvement from the baseline toward the expert reference. This gives useful feedback before correctness and performance-sensitive feedback after correctness. Fine-tuning Qwen3-32B with RLPF on PerfCodeBench raises correct-and-runnable solutions from $11.1\%$ to $54.6\%$ and improves relative efficiency from $8.1\%$ to $38.6\%$. The trained model becomes competitive with stronger open-weight systems, and its optimization behavior transfers modestly to EffiBench-X. Additional studies show that model-generated references provide useful but weaker supervision, and that the full composite reward is more reliable than correctness-only or runtime-only baselines. These results suggest that code agents can be trained not only to pass tests, but also to optimize the programs they write.

arXiv Semantic Scholar
View source

Similar papers

Preprint Jul 2026

Reinforcement Learning for Code Optimization

This work makes execution time learnable through three stages: how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations.

Pierre Chambon, Kunhao Zheng, Juliette Decugis et al. · 0 citations

Curriculum Sampling for RLOO Fine-Tuning on Countdown

This project investigates whether curriculum-based sampling can improve RLOO training by concentrating updates on prompts that are expected to be most informative and suggests a fundamental tradeoff between concentrating updates on potentially informative prompts and maintaining broad coverage of the training distribution.

Catherine Zhang, Nora Menon · 0 citations
Preprint Aug 2026

TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generation

Multi-turn code agents rely on execution feedback to repair incorrect programs, yet standard reinforcement learning paradigms optimize and evaluate policy performance primarily using single-shot outcome rewards. This misalignment conflates initial code generation with feedback-driven refinement, discards granular execution signals across intermediate turns, and fails to evaluate whether the policy actually acquires self-repair capabilities. We propose Test-aware Policy Refinement (TaPR), a framework that transforms execution feedback into a dense per-turn test-pass-ratio reward under a consistent multi-turn interaction protocol. Across six models on 219 code-generation problems from LiveCodeBench, TaPR improves the pooled three-turn success rate (Pass@3) by 2.44 percentage points. In the predefined 7B/8B high-headroom slice, pooled accuracy increases from 30.25% to 33.56% (+3.31 pp), with 42 improvements and 13 regressions in paired trials. On a matched Qwen3-8B ablation, the dense reward supplies nonzero feedback in all of the first ten steps and reaches a higher Hard-subset peak than outcome-only GRPO within the tested budget, although GRPO nearly matches pooled Pass@3 by step 300. Our primary contribution is a reward-decomposition framework and a turn-aware evaluation protocol that decouple first-shot generation quality from multi-turn repair competence.

Aofan Liu, Jing Meng, Fangxin Liu et al. · 0 citations

SFT-Estimated Curriculum Learning for Rule-Based RLOO Fine-Tuning

This project studies whether curriculum-based prompt ordering can make RL fine-tuning more stable and sample-efficient for language-model reasoning, and when curriculum structure helps, when it fails, and what failure modes appear in small-scale online RL fine-tuning.

Vanessa Felix · 0 citations
Conference Open access 2026

CodeRM-NT: Reward Model for Code RL without Unit Tests

This work proposes C ODE RM-NT, a code reward model with no reliance on unit tests that leverages Monte Carlo Tree Search guided by LLMs to generate code snippets and judges execution traces to annotate code with reward signals.

Xiao Xia, Dandan Zhang, Tian-Heng Sun · 0 citations