2025· Advances in Neural Information Processing Systems 38· 4 citations· 77 references
TL;DR
This work proposes LaRes, a novel hybrid framework that achieves efficient policy learning through reward function search by leveraging large language models to generate the reward function population, guiding RL in policy learning.
Abstract
The integration of evolutionary algorithms (EAs) with reinforcement learning (RL) has shown superior performance compared to standalone methods. However, previous research focuses on exploration in policy parameter space, while overlooking the reward function search. To bridge this gap, we propose LaRes , a novel hybrid framework that achieves efficient policy learning through reward function search. LaRes leverages large language models (LLMs) to generate the reward function population, guiding RL in policy learning. The reward functions are evaluated by the policy performance and improved through LLMs. To improve sample efficiency, LaRes employs a shared experience buffer that collects experiences from all policies, with each experience containing rewards from all reward functions. Upon reward function updates, the rewards of experiences are relabeled, enabling efficient use of historical data. Furthermore, we introduce a Thompson sampling-based selection mechanism that enables more efficient elite interaction. To prevent policy collapse when improving reward functions, we propose the reward scaling and parameter constraint mechanisms to efficiently coordinate reward search with policy learning. Across both initialized and non-initialized settings, LaRes consistently achieves state-of-the-art performance, outperforming strong baselines in both sample efficiency and final performance. The code is available at https://github.com/yeshenpy/LaRes .
Reinforcement learning holds significant potential for training large language models to handle multi-turn interactive tasks, but directly training with outcome rewards often results in slow convergence due to the sparsity of signals and the lack of fine-grained feedback.
Qiang Liu, Taian Guo, Ruizhi Qiao et al.· 0 citations
Designing an optimal reward function is fundamental to achieving stability and efficiency in reinforcement learning (RL). This is particularly critical in robotics, where sparse rewards often provide insufficient guidance, necessitating the inclusion of auxiliary state information to facilitate meaningful exploration. This work introduces Gradient Control Rewards (GCR), an interpretable, control-inspired reward-design methodology for accelerating agent training by modulating the reward signal based on the temporal dynamics of system error. Inspired by classical control theory, GCR partitions the reward into three distinct components, state alignment, bias correction, and dynamic stability. These elements synergistically discourage the accumulation of error and excessive velocity toward objectives, facilitating the acquisition of a well-regulated action policy. GCR was evaluated across diverse environments, ranging from simple pendulum simulations to high-fidelity robotic scenarios and external physical validation. Experimental results demonstrate that GCR achieves competitive performance compared to both conventional reward functions and adaptive methods such as Bootstrapped Reward Shaping (BSRS). While alternative approaches exhibit performance degradation in stochastic, real-world-representative simulations, GCR maintains robustness and has been successfully validated in external physical environments. These findings suggest that GCR offers a practical and interpretable framework for deploying RL in control-oriented physical systems.
Anas Aburaya, H. Selamat, M. Muslim et al.· IEEE Access· 0 citations
The proposed Module Level Reward Evolution Framework integrates three mechanisms: reflection-based refinement, hybrid credit assignment, and a merge strategy with rollback, which together improve the effectiveness and robustness of reward optimization.
Chenglin Liu, Xun Wang, Ruishuo Chen et al.· 0 citations
MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function across auxiliary tasks before RLHF training, is introduced, providing theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization.
Sparse, delayed, and weakly informative rewards remain central obstacles to efficient reinforcement learning. Reward shaping addresses these limitations by supplementing the task reward with an auxiliary signal that can accelerate learning while, in the classical setting, the original objective remains the evaluation criterion. Established theory guarantees safety for fixed shaping signals: potential-based reward shaping preserves optimal policies when the auxiliary term is the discounted difference of a time-invariant potential. In contemporary reinforcement learning systems, however, both the learner and the information available for guidance evolve during training: value estimates improve, novelty diminishes, feedback shifts, and predictive models are refined. Adaptive reward mechanisms occur across exploration, Bayesian inference, human-in-the-loop learning, automated reward design, and foundation-model-based approaches. This study introduces a unified analytical framework for comparing dynamic reward shaping and neighbouring adaptive reward mechanisms. The proposed framework distinguishes parametric revision from state-dependent variation, separates additive shaping from reward replacement and reward-adjacent guidance, and organises existing methods along temporal, informational, and theoretical dimensions. Using this framework, twelve method families are comparatively analysed. The framework further highlights the conditions under which optimality guarantees survive contemporary deep reinforcement learning pipelines, replay buffers, bootstrapped critics, and reward normalisation, while exposing the unresolved relationship between adaptation rate and learner stability.
