Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch

Abstract

Policy gradient methods are among the most successful approaches for solving challenging reinforcement learning problems. Despite their empirical success, many state-of-the-art policy gradient algorithms for discounted problems estimate update directions using a sampling distribution that differs from the discounted state distribution required by the policy gradient theorem. In this work, we analyze the impact of this distribution mismatch on policy gradient methods. Specifically, {for tabular direct and softmax parameterizations, we establish global convergence of objective values under biased updates}. {For tabular softmax policies, we further derive the exact leading-order coefficients of the asymptotic objective errors under biased and normalized exact updates. Their ratio provides an explicit criterion for determining when distribution mismatch accelerates or slows asymptotic convergence.} We then extend the analysis to more general policy parameterizations by deriving explicit bounds on both the state distribution mismatch and the induced normalized policy-gradient mismatch in episodic and continuing MDPs. These mismatches vanish at least linearly as the discount factor approaches one. Under an additional smoothness assumption, we further show that the limit inferior of the squared norm of the normalized exact policy-gradient direction along the biased iterates is bounded by $O((1-\gamma)^2)$ in both settings. Our results provide theoretical insights into the robustness of policy gradient methods to distribution mismatch and shed light on the gap between their theoretical formulations and practical implementations.