Natural and engineered safe reinforcement learning

Abstract

Abstract Survival depends not only on maximising rewards but on minimising harm. While Reinforcement Learning (RL) provides a normative framework for adaptive decision-making, standard algorithms often fail to account for the catastrophic costs of errors during learning, in the same way as animals do. In this review, we synthesise recent advances connecting the engineering of safe RL with behavioural and systems neuroscience. We adopt a principled taxonomy distinguishing between two strategies: (1) modifying the optimisation criterion to embed safety into the learning objective, and (2) modifying the exploration process to prevent dangerous actions. We show that engineering solutions, such as worst-case optimisation, risk-sensitive objectives (e.g., conditional value at risk, CVaR), and constrained Markov decision processes (MDPs), offer precise computational accounts for anxiety, risk-aversion, and defensive reflexes. Conversely, biological mechanisms, including innate priors, interoception, and meta-control based on uncertainty, reliability and competing needs, offer robust solutions to the problem of safe exploration in artificial agents. By bridging these domains, we highlight a converging computational landscape where safety is achieved through a dynamic interplay between reactive instincts and deliberative planning.