The Price of Failure Risk: Clipped Barrier Policy Gradients with Convex Regularization

Abstract

Many operations run on a budget that must rarely run out, such as a wear allowance. A wrapper keeps the remaining budget in the state, subtracts a barrier that grows as the budget shrinks and clips the combined reward from below at -m, and a standard policy-gradient learner trains unchanged. The clip level m is a price of failure risk, since the trained value splits exactly into utility before failure, the cost of a margin and the price times the risk, so whatever policy training returns earns the most such utility net of that cost among policies with no greater risk, up to its optimization error. Its risk exceeds a comparison policy's by at most order 1/m when charged costs stay bounded, and value or validation data certify it. The main theoretical result is exponential convergence of the population training flow under any parameter regularizer convex along the motions of the network's units and strong enough against the price, biased toward the prior policy. This extends mean-field convergence of policy gradients from relative entropy to weight decay, which covers continuous-time training of finite networks. Relative entropy adds noise that lets learning leave flat regions, weight decay pulls the parameters toward a reference, and action regularization bounds the risk change by the distance to a trusted rule. Experiments on an exact maintenance benchmark, a fuel-tank controller, a solvable flow and a benchmark-suite task measure the bounds' slack. We state when another method is better, without claiming this one outperforms them.