Constrained Residual Reinforcement Learning Augmentation of MRAC for Actuator-Safe Nonlinear Tracking

Abstract

Robust tracking under nonlinear uncertainty and actuator constraints remains challenging for model-reference adaptive control (MRAC). Deep reinforcement learning (RL) can compensate for nonlinear effects, but a standalone policy provides no inherent model-reference structure. This paper combines an MRAC backbone with a fixed, simulation-trained residual policy based on either Soft Actor–Critic (SAC) or Twin Delayed Deep Deterministic Policy Gradient (TD3). The combined command is limited in amplitude and slew rate before actuation. A Lyapunov analysis gives a conditional uniform ultimate boundedness characterization under the matching condition, bounded effective uncertainty, and bounded adaptive estimates; it explicitly includes the discrepancy introduced by actuator shaping. On the Quanser AERO 2 pitch-axis model, a 50-run Monte-Carlo study under severe parameter mismatch shows improved tracking relative to baseline MRAC without retuning. In hardware, the policy transfers from simulation without retraining; the TD3-based hybrid reduces pitch-tracking RMSE from 0.1057 to 0.0479 relative to MRAC, while standalone TD3 attains 0.0275. These results support a low-authority learned correction to MRAC within the tested single-axis operating conditions.