Time-Inconsistent MDPs with Entropy Regularization: Equilibrium Existence and Policy Iteration

Abstract

We study infinite-horizon time-inconsistent Markov decision processes with a countably infinite state space and unbounded reward functions. The reward is allowed to depend explicitly on the initial time and initial state, thereby accommodating general sources of time inconsistency. We seek relaxed feedback equilibria, and our approach is based on entropy regularization and weighted functional analytic methods. With entropy regularization, we characterize a regular relaxed equilibrium through a fixed-point operator. By introducing two weight functions with distinct roles, one controlling the growth of rewards and values and the other defining the ambient weighted space, we construct a compact invariant set under a product topology and apply the Schauder-Tychonoff fixed-point theorem to establish existence of regularized equilibria. Importantly, the invariant set can be chosen uniformly for small entropy weight $\lambda\in(0,1]$. We then let $\lambda\to0+$ and show, through compactness, concentration of Gibbs policies, and uniform-integrability arguments, that a subsequential limit is a relaxed equilibrium of the original unregularized problem. We further study a policy iteration algorithm (PIA) for the entropy-regularized equilibrium problem. Under a weighted-discounting structure and sufficiently strong discounting, we establish exponential convergence and uniqueness of the regularized equilibrium in a suitable weighted Banach space. Combining the policy-iteration error with a quantitative soft-max approximation bound, we show that the iterated policies constitute weighted $\varepsilon$-equilibria for the original unregularized problem and derive an explicit regret estimate. A numerical example illustrating the convergence of PIA under strong discounting and a counterexample demonstrating its failure under weak discounting are also provided.