PPO-Based Sim-to-Real Maples’ Navigation for TurtleBot3 Mobile Robots in Unknown and Dynamic Indoor Environments
TL;DR
The results of this study show that a PPO-based policy trained only in simulation and fine-tuned only on a small number of real-world tasks can compete with the classical and alternative DRL baselines in unknown and dynamic environments.
Abstract
Objective: In the field of mobile robotics, autonomous navigation in dynamic environments is one of the most challenging tasks in these environments: traditional methods based on pre-mapping and geometric planning are not effective in these environments due to uncertainty, and reactive methods are lacking in foresight. The work in this thesis tackles these issues by developing and testing an end-to-end Deep Reinforcement Learning (DRL) framework for maples navigation. Methods: A Proximal Policy Optimization (PPO) agent is trained using observations from LiDAR and goal-relative inputs in a high-fidelity open-source simulator that has been domain randomized in order to improve generalization. The trained policy is then transferred to a physical TurtleBot3 platform, with a safety supervisor controlled fine-tuning process. Comprehensive evaluation is performed on five simulated test scenarios (S=5), with 50 episodes per test scenario (250 episodes in total), with real-world trials performed on 20 trials in two different physical settings: cluttered lab and pedestrian corridor. The proposed approach is compared with a DDPG agent, as well as a standard A*+DWA pipeline using paired t-tests (α=0.05) and two-proportion z-tests, with statistical significance confirmed. Results: In simulation, the PPO agent has a success rate of 94% and a normalized path length of 1.18, both of which are significantly higher than those of A+DWA, which are 76% and 1.32 respectively (p<0.001 for both metrics); the agent takes 28.3 s to complete the task, which is significantly faster than A+DWA's 41.2 s (p<0.001). The success rate in the real-time laboratory test is 90%, and the inference time is 8.5ms/s. When the approach is used in structured corridor settings, the A+DWA baseline outperforms the PPO baseline with a 95% success rate vs. 85% for the PPO baseline, though. Novelty: The results of this study show that a PPO-based policy trained only in simulation and fine-tuned only on a small number of real-world tasks can compete with the classical and alternative DRL baselines in unknown and dynamic environments. The results form a feasible basis for the implementation of a learning-based navigation controller on low-cost mobile platforms and offer a fair comparison between the performance and limitations of learning-based navigation controllers and traditional ones.