Shaping Gradient and Exploration-Noise Initialization, Not Reward Polarity, Determine Convergence in Deep Reinforcement Learning for Autonomous Quadrotor Navigation and Obstacle Avoidance
TL;DR
It is concluded that a continuous shaping gradient and the exploration-noise initialization, interacting with the curriculum advancement criterion, determine convergence in continuous control deep reinforcement learning, and that reward polarity by itself does not.
Abstract
This paper presents a systematic reward engineering methodology for training a Proximal Policy Optimization (PPO) quadrotor navigation policy in the Webots simulator, using a hierarchical architecture in which a PID controller handles low-level stabilization and a PPO policy issues velocity commands. We document the complete evolution of a composite ten-term reward function across seven versions (v5 through v11) and retrain the key versions with multiple independent training seeds. The multi-seed study revises the single-seed history: penalty-dominated configurations (v8, v10) fail across all seeds, while the strongest historical version proves seed-sensitive (v11: 32.2 +/− 15.8%). An ablation removing the continuous distance-shaping term from v11 yields 0% success across seven seeds, identifying that term as necessary for convergence. We further isolate a previously hidden co-factor: with the library-default exploration-noise initialization (sigma_0 = 1.0), sampled actions saturate the bounded action space, the exploration variance receives no learning gradient, and curriculum progression deadlocks regardless of reward design; initializing sigma_0 = 0.37 restores gradient flow. With this correction and a deterministic evaluation-gated curriculum, the final configuration is evaluated across the full curriculum rather than at a single operating point: across five independent training seeds under a deterministic protocol, it attains 95.0% ± 6.2% navigation success at Stage 0 conditions (2 m targets, no obstacles), 89.6% ± 6.9% at Stage 1 conditions (4 m, one obstacle), and 48.4% ± 10.8% at Stage 2 conditions (7 m, three obstacles). Reporting this difficulty curve, rather than a single headline value, exposes a substantial generalization gap whose dominant failure mode is obstacle collision (45–52% of episodes at Stage 2). Matched retraining of Soft Actor-Critic and TD3 baselines under identical reward and curriculum conditions yields one completed seed each both baselines show non-monotonic difficulty curves, and at Stage 2 conditions, TD3 (64.0%) exceeds PPO (48.4% ± 10.8%) while SAC (43.0%) falls just below it, whereas at Stage 0, PPO (95.0%) leads both, so the ranking is operating point-dependent on the current single-seed evidence. We conclude that a continuous shaping gradient and the exploration-noise initialization, interacting with the curriculum advancement criterion, determine convergence in continuous control deep reinforcement learning, and that reward polarity by itself does not.