On the Optimality of Markovian Policies for Chance-Constrained Covariance Steering
TL;DR
It is shown that, even for the covariance steering problem with a broad class of commonly used state and control safety constraints, the synthesized Markovian policy almost surely produces the same control actions as the history-dependent policy and therefore the same state trajectories, cost, and moments.
Abstract
Many studies on finite-horizon stochastic optimal control, including covariance steering, parameterize control policies as state-history-affine. This parameterization enables a convex reformulation, thereby yielding a tractable solution method. However, the necessity of dependence on previous states has not been well established. \textit{Is this dependence necessary, or merely an artifact of the convex reformulation?} We show that it is an artifact that can be removed losslessly. Given an optimal solution of the state-history-affine formulation, we construct a deterministic Markovian policy which is affine in the current state. We show that, even for the covariance steering problem with a broad class of commonly used state and control safety constraints, the synthesized Markovian policy almost surely produces the same control actions as the history-dependent policy and therefore the same state trajectories, cost, and moments. Thus, every optimum of the history-dependent formulation admits a lossless Markovian transformation. Geometrically, the history-dependent formulation lifts the policy space for convexity, and its optimal solution can be projected back to the Markovian policy space. We extend the analysis to output feedback and a convex upper-bounding surrogate for value-at-risk costs.