Safe Reinforcement Learning under Regularized Probabilistic Counterexample Guidance

Persistent Safety Set Guided Offline Safe Reinforcement Learning

Directional Constraints for Efficient Exploration in Safe Reinforcement Learning

Reinforcement Learning has revolutionized the landscape of robotic research, allowing robust learning of complex robotic skills in simulation. However, real-world deployment in open-ended environments requires strong safety guarantees to prevent dangerous or harmful behaviors. Safe Reinforcement Learning methods address this requirement by enforcing safety constraints. Nevertheless, learning under constraints often reduces learning speed and could lead to suboptimal task performance, as the agent must solve a more complex constrained optimization problem compared to unconstrained settings. To tackle this issue, in this work, we propose an extension of the ATACOM framework, a state-of-the-art reliable safety layer that can be integrated with existing Reinforcement Learning algorithms to enforce constraints derived from prior knowledge of the system or learned directly from data. Our proposed method, named ATACOM Directional Constraints (ATACOM-DC), significantly improves the safety-performance trade-off by introducing directional constraints that distinguish between actions approaching and moving away from constraint boundaries, activating constraint enforcement only when necessary. We evaluate our method across a range of challenging robotic control tasks in simulation, analyzing both constraint-violation costs and achieved task performance. Code and additional material at https://atacom-dc.robot-learning.net.

Certifiable Safe Model-Based Reinforcement Learning with Control-Affine Dynamics Approximation

Safe model-based reinforcement learning (RL) often bridges control-theoretic analysis and RL for robots to safely explore (partially) unknown system dynamics while deriving control actions for task efficiency. The control performance and safety assurance typically rely on prior knowledge of partially modeled nominal system dynamics and the data-driven models that compensate for residual model uncertainties. However, existing methods often overlook the structure of residual model uncertainties (e.g., components affine in control), which could lead to overly conservative robot behaviors or invalid safety guarantees under the safe learning-based controllers. This paper proposes a safe reinforcement learning framework that learns control-affine dynamics with a certifiable data-driven safe policy using control barrier functions (CBF). Specifically, we first use Control-Affine Random Fourier Features (ARFF) to model robot dynamics in a control-affine form, which offers computational efficiency that scales with dataset size and reduces potential model bias for model-based reinforcement learning. Then, a model-free, efficient uncertainty quantification method using adaptive conformal prediction (ACP) is applied to quantify the uncertainty in the safety constraint arising from the learned control-affine dynamics. This allows for data-driven safety assurance amenable to principled and efficient controller synthesis with CBF. Simulation results on the cartpole and the 3D quadrotor platforms demonstrate the effectiveness of the proposed framework.

Safe Reinforcement Learning using Ideas from Model Predictive Control

Reinforcement learning (RL) enables the synthesis of control policies directly from data, making it highly appealing for complex cyber-physical systems (CPSs) and robotics. A persistent challenge, however, is ensuring strict, hard safety constraints during the active learning phase. In real-world physical systems, violating mechanical limits can cause irreversible damage, necessitating that exploration remains strictly within safe operational regions. We propose a generalized framework that combines the adaptive, high-performance nature of deep reinforcement learning (DRL) with the formal safety guarantees of model predictive control (MPC). Using a mathematical model of the system dynamics, offline MPC computations define a feasible state-action space, representing all safe combinations of system states and control inputs that guarantee constraint satisfaction. During training and deployment, the RL agent's instantaneous actions are projected onto this globally verified feasible set via a safety filter. We systematically evaluate our generalized approach on a non-linear 1-DoF laboratory testbed, demonstrating successful exploration and stable policy convergence on physical hardware.

DynBudget: Adaptive Risk Calibration and Dynamic Budgeting for Safe Reinforcement Learning Under Distribution Drift

Ensuring safety constraint satisfaction in non-stationary Reinforcement Learning (RL) environments remains a major challenge, as distribution drift can quickly invalidate previously calibrated risk estimates. To address this issue, we propose DynBudget, a closed-loop Safe RL framework integrating a learned safety critic, temperature-calibrated risk estimation, and a dynamic safety budget. By jointly adjusting the calibration temperature and the intervention threshold based on real-time violation feedback, the safety shield enables rapid adaptation to environmental drifts. We carried out experiments in two stages to fully evaluate the adaptation mechanism as well as the practical robustness of DynBudget. First, we studied the framework in a number of discrete MiniGrid contexts, and then we tested it on various continuous robotic tasks in Safety-Gymnasium. Our empirical results indicate that our approach is able to effectively reduce safety violations, shorten recovery time after a drift, and preserve competitive task performance when faced with abrupt changes in hazards or physical perturbations. Finally, we show that shielding with dynamic budgets is an interpretable and viable approach to Safe RL in autonomous systems.

Evaluating Fuzz Testing for Reinforcement Learning Agents

It is shown that fuzzing-generated crashes can meaningfully improve agent robustness and enable accurate safety monitoring with strong cross-method generalization, and the benefits of combining complementary fuzzing strategies and adopting multi-level diversity analysis to achieve more comprehensive and practical RL testing.