PD-SRL: Parallel and Differentiable Surgical Simulation For Accelerated Reinforcement Learning
TL;DR
PD-SRL is introduced, the first differentiable simulator tailored for RL-driven automation of surgical tasks and allows the use of differentiable RL methods, such as SHAC and BPTT, to accelerate training compared with consolidated RL techniques such as PPO.
Abstract
The advancement of surgical robotics has the potential to standardize surgical workflows and reduce clinician workload by assisting with repetitive tasks. Reinforcement Learning (RL) offers an approach to train robotic agents for surgical tasks, such as soft tissue manipulation, although this comes at the cost of requiring large amounts of training data. Several surgical simulators have therefore been introduced to facilitate RL training data generation, yet advanced simulation features often come at the expense of training efficiency. In this work, we introduce PD-SRL, the first differentiable simulator tailored for RL-driven automation of surgical tasks. PD-SRL runs multiple complex soft-body simulations concurrently while providing first-order gradients. Thus, our simulator allows the use of differentiable RL methods, such as SHAC and BPTT, to accelerate training compared with consolidated RL techniques such as PPO. We validate the simulator through experiments on simulated surgical tasks of increasing complexity and show that policies trained in simulation can be deployed on a real da Vinci Research Kit setup. PD-SRL reduces wall-clock convergence time by 50% to over 95% compared to an RL baseline without compromising the quality of the policy obtained.