Reward Shaping based on Trajectory Quality for offline and hybrid reinforcement learning

Abstract

Current offline reinforcement learning algorithms typically apply equal learning strength to samples of different quality during training, which can limit the effective utilization of high-quality trajectories. To solve this problem, we propose Reward Shaping based on Trajectory Quality (RSTQ). RSTQ increases the rewards of high-quality samples, encouraging the agent to better learn their characteristics and thereby enhancing overall performance. RSTQ is integrated with TD3BC and CQL, and is systematically evaluated on the D4RL benchmark. Experimental results show that RSTQ significantly improves performance on Gym-MuJoCo tasks, with the average normalized returns of TD3BC and CQL increased by 14.2% and 21.6%, respectively, and performance gains exceeding 60% on several tasks, achieving strong competitive results on the evaluated offline reinforcement learning benchmarks. Moreover, RSTQ effectively extends to online reinforcement learning with offline datasets, where it also achieves strong performance.