Application and evaluation of reinforcement learning for two-dimensional trajectory tracking in snake-like robots

Abstract

Context—Snake-like robots are biomimetic systems that can move effectively in narrow, complex, and restricted environments thanks to their modular and flexible body structures composed of numerous serially connected joints. These characteristics offer significant advantages, particularly in areas such as pipeline inspection, search and rescue operations, industrial maintenance applications, and exploration missions. The multiple degrees of freedom distributed along the body enable the robot to achieve high maneuverability but also make the control problem quite complex. Due to the dynamic interactions between segments, friction-based motion characteristics, and nonlinear system behavior, achieving reliable and accurate trajectory tracking emerges as a significant engineering problem.Objective—In this study, a reinforcement learning (RL) based control method has been developed to solve the trajectory tracking problem for snake-like robots in a two-dimensional plane.Method—In the proposed approach, the robot’s dynamic model was created in the Webots simulation environment, an open-source simulation program, and all training and testing processes were carried out in this environment. During the learning process, policy- based RL algorithms from the Stable-Baselines library were used. In this context, Proximal Policy Optimization (PPO) and three different RL algorithms were used during the training process. To enable the robot to adapt to different orientation scenarios, seven different angles defined in the range of +45 to −45 and trajectories of varying lengths were used. Thus, the goal was for the agent to learn a generalizable control policy not only for a specific trajectory type but also for tracks with different slopes and orientations.Results—The results obtained show that the PPO algorithm produced a higher average reward compared to other methods and exhibited a more stable learning process. After training was completed, the developed method was tested both on trajectories used during the training phase and on previously unseen trajectories. For the 0 trajectory, maximum errors were recorded as 0.093 m and 0.040 m for the x and y axes, respectively. Furthermore, the system exhibited robust generalization capabilities on a +22.5 trajectory, not encountered during the training phase, yielding maximum errors of 0.099 m and 0.052 m.Conclusion—These findings demonstrate that the proposed RL-based control approach can effectively solve the two-dimensional trajectory tracking problem in snake robots. In future studies, the proposed method can be extended to the three-dimensional trajectory tracking problem, or it can be evaluated under more complex conditions, such as scenarios involving obstacles.