Unmanned Underwater Vehicle Path Following Based on Improved Soft ActorCritic with Action Smoothing

Abstract

The deep reinforcement learning (DRL) method shows great potential in unmanned underwater vehicle (UUV) motion control. Traditional control methods rely on an accurate UUV model. However, due to the nonlinearity of the UUV model and the complex underwater environment, it is difficult to obtain an accurate model in practice. Therefore, we propose a DRL-based path-following control method for UUV. Specifically, this method integrates the Soft Actor–Critic (SAC), prioritized experience replay (PER), and policy smoothing methods to improve the generalization, tracking accuracy, and smoothness of the controller. PER assigns greater sampling weights to samples with high learning efficiency. Another major challenge in the UUV control problem arises from the action oscillations induced by the DRL policy. This problem leads to excessive wear of the actuator system and difficulty in practical applications. To address this, an improved policy regularization method is designed to promote action smoothness. In addition, a dynamic reward function that integrates tracking error, motion state, and control signals is designed to improve training efficiency. The simulation results show that improved SAC has a smooth control signal and good tracking accuracy in different waypoint configurations and uncertain disturbances.