A Comparative Study of Reinforcement Learning Algorithms Under Task Performance and Energy Constraints with Transfer Learning Analysis

Abstract

Reinforcement learning (RL) is a method of training artificial intelligence agents to make decisions through trial and error, rewarding good behavior and penalizing bad behavior until the agent learns an effective strategy. This study compares three widely used RL algorithms for continuous robotic control: Proximal Policy Optimization (PPO), which learns by making small, cautious updates to avoid unstable training; Soft Actor-Critic (SAC), which encourages broad exploration by rewarding the agent for trying diverse strategies alongside completing the task; and Advantage Actor-Critic (A2C), which uses two neural networks simultaneously - one to decide actions and one to evaluate them. Three experimental conditions were evaluated across three robotic simulation environments of increasing difficulty: standard task performance, energy-aware performance (where agents were equally penalized for excessive energy use alongside task completion), and transfer learning (where agents pre-trained on standard rewards were fine-tuned on energy-aware rewards). SAC consistently outperformed PPO and A2C on dense-reward environments under both standard and energy-constrained conditions. Effect sizes indicated some nonsignificant differences were nonetheless practically large. Most strikingly, A2C exhibited highly unstable performance under energy-aware rewards on MountainCarContinuous-v0, achieving a mean reward of -18,231 compared to near-zero values for PPO and SAC, suggesting that A2C may require additional reward shaping or tuning before use in energy-constrained robotic applications. Transfer learning experiments revealed that pre-training on standard rewards generally hurt rather than helped adaptation to energy-aware objectives. These findings offer practical guidelines for algorithm selection in energyconstrained robotic systems such as prosthetic hands and autonomous vehicles.