Exploration and Exploitation: A Study on Sample Efficiency in Reinforcement Learning With Multifaceted Curiosity Rewards and Adaptive Experience Replay Utilisation in Sparse Reward Environments
Jul 2026· CAAI Transactions on Intelligence Technology· Vol 11, pp. 1028-1046· 0 citations· 14 references
TL;DR
A reinforcement learning framework built upon the Soft Actor Critic architecture, which integrates multifaceted curiosity rewards (MCR) and adaptive experience replay utilisation (AERU) (MCR‐AERU SAC), demonstrating superior sample efficiency and excellent robustness in large‐scale sparse reward environments.
Abstract
In recent years, the widespread application of deep reinforcement learning (DRL) in autonomous systems has highlighted the importance of achieving high sample efficiency under sparse reward conditions. To improve sample efficiency in sparse reward environments, this paper proposes a reinforcement learning framework built upon the Soft Actor Critic architecture, which integrates multifaceted curiosity rewards (MCR) and adaptive experience replay utilisation (AERU) (MCR‐AERU SAC). MCR combines multi‐level intrinsic motivational signals, such as state prediction error and model uncertainty, to provide rich exploration incentives, encouraging the controlled entity to deviate from existing trajectories and discover new high‐reward behaviours. AERU dynamically adjusts the experience replay priorities based on posterior temporal difference error (TD error), focusing on utilising ‘partially successful’ transitions that are easily overlooked. The synergy between MCR and AERU enables the proposed framework to achieve an optimal balance between exploration and exploitation, significantly accelerating policy convergence and improving sample utilisation. Extensive experiments in complex dynamic training environments demonstrate that the proposed MCR‐AERU SAC algorithm achieves up to 1.41 times the early‐stage gain rate and a 55.56% improvement in task success rate compared to the HER‐SAC baseline, demonstrating superior sample efficiency and excellent robustness in large‐scale sparse reward environments.
This work proposes LaRes, a novel hybrid framework that achieves efficient policy learning through reward function search by leveraging large language models to generate the reward function population, guiding RL in policy learning.
Pengyi Li, Hongyao Tang, Jinbin Qiao et al.· Advances in Neural Informati...· 4 citations
The results demonstrate the potential of adaptive reward weighting to provide a systematic mechanism for controlling the exploration–imitation trade-off and enhancing the stability and robustness of GAIL-based policy learning while retaining the exploration advantages of the TD3-SAC hybrid framework.
Mehran Ali, Zia Ullah, Aliza Ashfaq· Journal of Engineering and C...· 0 citations
Designing an optimal reward function is fundamental to achieving stability and efficiency in reinforcement learning (RL). This is particularly critical in robotics, where sparse rewards often provide insufficient guidance, necessitating the inclusion of auxiliary state information to facilitate meaningful exploration. This work introduces Gradient Control Rewards (GCR), an interpretable, control-inspired reward-design methodology for accelerating agent training by modulating the reward signal based on the temporal dynamics of system error. Inspired by classical control theory, GCR partitions the reward into three distinct components, state alignment, bias correction, and dynamic stability. These elements synergistically discourage the accumulation of error and excessive velocity toward objectives, facilitating the acquisition of a well-regulated action policy. GCR was evaluated across diverse environments, ranging from simple pendulum simulations to high-fidelity robotic scenarios and external physical validation. Experimental results demonstrate that GCR achieves competitive performance compared to both conventional reward functions and adaptive methods such as Bootstrapped Reward Shaping (BSRS). While alternative approaches exhibit performance degradation in stochastic, real-world-representative simulations, GCR maintains robustness and has been successfully validated in external physical environments. These findings suggest that GCR offers a practical and interpretable framework for deploying RL in control-oriented physical systems.
Anas Aburaya, H. Selamat, M. Muslim et al.· IEEE Access· 0 citations
Novelty and Surprise Prioritized Experience Replay (NSPER) is introduced, which uses novelty to capture underrepresented states and surprise to expose gaps in the agent's understanding of the environment and is extended with NSPER+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration.
Hoda Yamani, Henry Williams, Bruce A. MacDonald· 0 citations
Sparse, delayed, and weakly informative rewards remain central obstacles to efficient reinforcement learning. Reward shaping addresses these limitations by supplementing the task reward with an auxiliary signal that can accelerate learning while, in the classical setting, the original objective remains the evaluation criterion. Established theory guarantees safety for fixed shaping signals: potential-based reward shaping preserves optimal policies when the auxiliary term is the discounted difference of a time-invariant potential. In contemporary reinforcement learning systems, however, both the learner and the information available for guidance evolve during training: value estimates improve, novelty diminishes, feedback shifts, and predictive models are refined. Adaptive reward mechanisms occur across exploration, Bayesian inference, human-in-the-loop learning, automated reward design, and foundation-model-based approaches. This study introduces a unified analytical framework for comparing dynamic reward shaping and neighbouring adaptive reward mechanisms. The proposed framework distinguishes parametric revision from state-dependent variation, separates additive shaping from reward replacement and reward-adjacent guidance, and organises existing methods along temporal, informational, and theoretical dimensions. Using this framework, twelve method families are comparatively analysed. The framework further highlights the conditions under which optimality guarantees survive contemporary deep reinforcement learning pipelines, replay buffers, bootstrapped critics, and reward normalisation, while exposing the unresolved relationship between adaptation rate and learner stability.