Fuzzy Unified Iteration With Bilevel Optimization for Unknown Multiagent Systems

Abstract

In multiagent systems (MASs), policy iteration (PI) and value iteration (VI) are two fundamental reinforcement learning paradigms for solving optimal cooperative control problems, exhibiting an inherent tradeoff between convergence speed and communication complexity. To clarify this tradeoff, this article reformulates the Bellman optimality equation for MASs from a bilevel optimization perspective and rigorously establishes its equivalence to the original problem. The resulting iterative solution consists of two coupled levels for the value function and control policy. By examining how value function initialization affects the existence of zero-residual solutions of Bellman equations, a unified iteration (UI) paradigm is derived. The resulting paradigm inherits the fast convergence of PI while retaining the low communication complexity of VI. The impact of suboptimal solutions on the convergence of iterative value functions and the effect on the admissibility of the iterative control policies are examined. To implement the UI paradigm in practice, a batch data-driven fuzzy identifier–actor–critic structure is proposed, and the closed-loop stability of the learning system is established. The effectiveness of the proposed method is demonstrated on a benchmark problem.