PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities.
Abstract
Reinforcement learning (RL) has recently emerged as a promising feedback control strategy for nonlinear and complex dynamical systems. However, RL algorithms are sample inefficient and require a large number of interaction with the environment to synthesize optimal control strategies. Consequently, applications of RL are typically limited to sparse sensors and actuators due to the curse of dimensionality entailed by the exploration-exploitation dilemma in high-dimensional spaces. In this work, we bridge RL and traditional optimal control for dynamical system with a novel Physics-EnhAnced Reinforcement Learning (PEARL) paradigm tailored to the control of high-dimensional and parametric dynamical systems, exploiting the differentibility of their dynamics. Specifically, PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities. Through two challenging parametric navigation problems in unsteady flows, we show that PEARL (i) effectively exploits differentiable environments to outperform state-of-the-art RL algorithms, (ii) is sample efficient, thanks to the physics-guided policy learning, (iii) generalizes across multiple scenarios, which is crucial when dealing with parametric systems, and (iv) enables scaling RL to high-dimensional state and action spaces, without requiring low-dimensional state representations or multi-agent strategies.
This work uses SHallow REcurrent Decoder networks-based Reduced Order Modeling (SHRED-ROM) to synthesize a real-time closed-loop controller for high-dimensional and parametric dynamics, relying solely on limited state sensor readings, alleviating the curse of dimensionality.
Matteo Tomasetto, Francesco Braghin, J. Kutz et al.· 0 citations
Stochastic reach-avoid analysis of controlled dynamical systems is an important tool for safety-critical control under uncertainty, in which the reach-avoid probability is characterized by a Hamilton-Jacobi partial differential equation (PDE). However, solving this PDE using conventional numerical methods becomes computationally intractable as the system dimension increases. Physics-informed neural networks (PINNs) may converge to inaccurate local minima when trained primarily through PDE-residual minimization. Reinforcement learning (RL) offers a scalable alternative, but its learned value functions may be inaccurate or inconsistent with the governing PDE. This paper proposes a physics-informed RL (PIRL) framework that combines the complementary strengths of PINNs and RL for stochastic reach-avoid analysis. We develop a scheduled PIRL algorithm in which temporal-difference actor-critic learning first guides the critic toward a meaningful approximation of the reach-avoid value function. PDE-residual and boundary-condition losses are then introduced progressively to enforce consistency with the governing PDE and its boundary conditions. The proposed method mitigates the failure modes of conventional PINN techniques while achieving accuracy comparable to that of successfully trained PINNs. The effectiveness of the proposed framework is demonstrated through two case studies.
This work introduces a constrained heuristic-learning protocol in which an agent iteratively proposes, evaluates, and revises controller implementations while interacting exclusively through the public benchmark interface, and suggests that heuristic learning through coding agents constitutes a credible and complementary alternative to conventional reinforcement learning.
A reinforcement learning approach using the TD3 algorithm to train autonomous agents to reach arbitrary targets within a parametric, chaotic double-gyre flow is presented and a trade-off in sensor utility is revealed: velocity-aware agents optimize energy efficiency, whereas vorticity sensors provide superior structural mapping and achieve better target proximity.
Andrea Braghin, Nicolò Botteghi, Matteo Tomasetto et al.· 0 citations
Designing an optimal reward function is fundamental to achieving stability and efficiency in reinforcement learning (RL). This is particularly critical in robotics, where sparse rewards often provide insufficient guidance, necessitating the inclusion of auxiliary state information to facilitate meaningful exploration. This work introduces Gradient Control Rewards (GCR), an interpretable, control-inspired reward-design methodology for accelerating agent training by modulating the reward signal based on the temporal dynamics of system error. Inspired by classical control theory, GCR partitions the reward into three distinct components, state alignment, bias correction, and dynamic stability. These elements synergistically discourage the accumulation of error and excessive velocity toward objectives, facilitating the acquisition of a well-regulated action policy. GCR was evaluated across diverse environments, ranging from simple pendulum simulations to high-fidelity robotic scenarios and external physical validation. Experimental results demonstrate that GCR achieves competitive performance compared to both conventional reward functions and adaptive methods such as Bootstrapped Reward Shaping (BSRS). While alternative approaches exhibit performance degradation in stochastic, real-world-representative simulations, GCR maintains robustness and has been successfully validated in external physical environments. These findings suggest that GCR offers a practical and interpretable framework for deploying RL in control-oriented physical systems.
Anas Aburaya, H. Selamat, M. Muslim et al.· IEEE Access· 0 citations