Dual-critic constrained deceptive Q-learning for deployment-time policy protection

Abstract

Securing the decision-making process of reinforcement learning (RL) agents during deployment is crucial in privacy-sensitive and security-critical domains. However, as deployed policies interact with the environment, they generate observable trajectories that can inadvertently leak sensitive decision patterns. External observers can readily exploit these trajectories via trajectory-based attacks such as imitation learning (IL), inverse reinforcement learning (IRL) to extract the policy or infer the underlying reward structure. Current deployment-time defenses typically tackle policy extraction or reward inference in isolation, and often fail to provide strict guarantees on the agent’s task utility. To bridge this gap, we propose the Dual-Critic Constrained Deceptive Q-Learning (DCD-Q) method, a deployment-time trajectory protection framework that aims to reduce the information leaked by released trajectories while preserving acceptable task performance. DCD-Q employs a utility critic to constrain decisions to a near-optimal candidate action set, aiming to preserve task utility while enabling controlled deceptive behavior. Simultaneously, it constructs a dynamic anti-reward from sliding-window visitation statistics to bias execution toward less-visited feasible actions. This induces a controlled, time-varying deceptive behavior in the released trajectories. We theoretically analyze the utility-preserving component by deriving a lower bound on the expected return under constrained action selection. Experiments on benchmark environments evaluate DCD-Q against representative trajectory-based attackers, which show that DCD-Q reduces the effectiveness of policy and reward recovery while maintaining a task-dependent utility–protection trade-off.