Canon, a risk-averse method designed to optimize Value-at-Risk (VaR) constrained reinforcement learning (RL) problems, is introduced and Cantelli's inequality is employed to obtain a tractable, conservative and smooth bound on the VaR constraint based on the first two moments of the cost return.
Abstract
We introduce Canary, a risk-averse method designed to optimize Value-at-Risk (VaR) constrained reinforcement learning (RL) problems. We employ Cantelli's inequality to obtain a tractable, conservative and smooth bound on the VaR constraint based on the first two moments of the cost return. This yields a constraint estimator that remains stable with tight violation thresholds in dense cost regimes. Extending the trust-region framework of the Constrained Policy Optimization (CPO) method, we further provide worst-case bounds for both policy improvement and constraint violation during the training process. Empirically during training, Canary is the only method that reliably satisfies the VaR constraint in every environment tested.
The per-iteration program is solved with sequential convex programming (SCP), the standard solver for difference-of-convex problems, and convergence guarantees under mild conditions are given, bridging the difference-of-convex optimization and RL literatures.
Shripad Deshmukh, Yaswanth Chittepu, Dhawal Gupta et al.· 0 citations
This work exploits the auxiliary-threshold representation of CVaR to establish the existence of an optimal strategy and strong duality without requiring market completeness, and proves that the resulting strategies converge to the optimal control as the number of iterations tends to infinity.
An-Ran Hu, Silvana M. Pesenti, Xiaofei Shi· 0 citations
BCPPO (Bachelier-Inspired Constrained Proximal Policy Optimization), a proximal policy optimization (PPO) method, supports a practical balance among reward, caution around cost predictions that vary across trained critics, and policy-only deployment.
Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent, is introduced and it is proved that it has the same unique optimal solution as the original PMD objective.
Safe reinforcement learning (RL) commonly enforces expected-cost constraints, but such expectation safety may fail to control the probability of rare high-cost trajectories. Chance-constrained MDPs (CCMDPs) impose a stronger probability-level requirement, but are widely viewed as harder because the chance constraint is...
Decision-focused learning for linear optimization is complicated by the discontinuity of the optimizer, where small cost errors may leave the decision unchanged or move it to a different vertex. We show that this non-smooth pointwise behavior becomes locally quadratic after averaging over the data distribution, and we...
Konstantinos Ziliaskopoulos, A. Vinel, Alice E. Smith· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 6, 2026
A machine-learning approach called CheMeleon provides a practical way to develop new drugs or other molecules with much less detailed, expensive experimentation.
Learn how Google DeepMind and GovTech Singapore uses lightweight on-device Gemma models to intercept conversational scams in private chats with zero cloud data egress.
