2026· Annual Meeting of the Association for Computational Linguistics· pp. 43316-43333· 0 citations· 37 references
Computer Science
TL;DR
This work proposes C ODE RM-NT, a code reward model with no reliance on unit tests that leverages Monte Carlo Tree Search guided by LLMs to generate code snippets and judges execution traces to annotate code with reward signals.
Abstract
Providing accurate reward signals for code generated by large language models (LLMs) is a significant challenge in applying reinforcement learning (RL) to code generation. Existing methods rely on unit tests to evaluate code correctness and provide rewards, which are hindered by the difficulty of acquiring and verifying reliable unit tests at scale. In this work, we propose C ODE RM-NT, a code reward model with no reliance on unit tests. Our method leverages Monte Carlo Tree Search guided by LLMs to generate code snippets and judges execution traces to annotate code with reward signals. We use the rewards to train C ODE RM-NT that is capable of providing rewards for code during RL. C ODE RM-NT also facilitates curriculum learning by scoring and sorting training samples based on their difficulty. Experimental re-sults demonstrate that training with C ODE RM-NT consistently outperforms synthetic unit test-based rewards, yielding superior performance on multiple code generation benchmarks. Additionally, curriculum learning based on C ODE RM-NT further enhances model performance. Our code and dataset are available at: https://github.com/THUDM/CodeRM-NT .
This work proposes RLPF, reinforcement learning from performance feedback, which turns execution outcomes into a staged reward, and suggests that code agents can be trained not only to pass tests, but also to optimize the programs they write.
Huihao Jing, Haozhe Cui, Wenbin Hu et al.· 0 citations
This work makes execution time learnable through three stages: how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations.
Pierre Chambon, Kunhao Zheng, Juliette Decugis et al.· 0 citations
Reinforcement Learning from Verifiable Rewards (RLVR) is pivotal for enhancing LLM code generation, yet its efficacy is often hindered by insufficient test case coverage, leading to reward hacking and policy degradation. To address this, we propose RobustTests, a framework featuring a faulty-code-driven test case synthesis strategy. By leveraging"near-correct"faulty codes, RobustTests captures latent logical discrepancies and employs validator agents with behavioral feature clustering to filter invalid or redundant test cases. Additionally, a stepwise dense reward function based on pass rates is introduced to mitigate false negatives and enhance training robustness. Using this pipeline, we construct an augmented version of the CodeContests+ dataset with superior diagnostic utility. Experimental results show that RL fine-tuning of Qwen3-32B via RobustTests achieves a 3% absolute gain on LiveCodeBench, demonstrating its effectiveness in advancing LLM code generation proficiency. Codes and data are available at https://huggingface.co/datasets/sid6/RobustTests.
Yiwen Zhang, Xiaodong Yan, Zhenyu Huang et al.· 0 citations
Empirical findings suggest that CudaPerf significantly outperforms strong baselines, including Qwen-3-32B and CUDA Agent by achieving up to 5X and 3.32X improvements in speedup, and 17%&7% improvements in correctness, respectively.
Q. I. Mahmud, Nesreen K. Ahmed, Ali Jannesari· 0 citations
DHRCL decomposes feedback into syntax validation, execution success, unit-test pass rate, and AST-based structural similarity, and organizes these signals through a three-stage Syntax, Execution, Pass&Structural curriculum, and introduces stage-aware probability-based token credit redistribution.
In this paper, we consider the setting where large language models (LLMs) are trained using reinforcement learning (RL) to simultaneously improve reasoning accuracy and verbalize its confidence. Our reward scheme uses two functions for rewarding confidence verbalized by the LLM: one when the LLM is correct and a different one when the LLM is incorrect. With a poorly designed reward scheme, the LLM may be incentivized to answer incorrectly so that it can be confident that its answer is indeed incorrect, a phenomenon that we call confidence reward hacking. We propose the concept of non-hackable confidence reward schemes and define a spectrum of such reward schemes for RL confidence calibration training in LLMs. We demonstrate that selective confidence reward hacking can occur in practical datasets with reward schemes that are not designed to be non-hackable. We also demonstrate that the reward scheme with the best calibration to accuracy tradeoff depends on the dataset and the application, and propose using the reward scheme as a hyperparameter to optimize the tradeoffs in accordance to what is important for the application. The code of our experiments is available in https://anonymous.4open.science/r/rl-confidence-calibration-9ED4/README.md.
Chee Heng Tan, Zhuoyi Lin, M. Motani et al.· 0 citations