Skip to content
Preprint

Bayesian Offline Multi-Agent Reinforcement Learning with Latent World Models

Oct 2026 · 0 citations · 47 references
Computer Science

TL;DR

A Bayesian offline MARL framework that infers latent global states from decentralized observation-action histories and iteratively refines a posterior world model using only offline data is proposed, supporting a principled connection between world model reliability, offline data coverage, and policy performance in partially observable offline MARL.

Abstract

Model-based offline multi-agent reinforcement learning (MARL) provides a promising approach to multi-agent decision-making under partial observability, where the global state is unavailable from offline data. However, limited dataset coverage can lead to substantial out-of-distribution (OOD) model errors, which may mislead policy optimization and degrade downstream performance. Existing model-based offline MARL methods typically rely on access to global states or mitigate model errors through pessimistic or conservative policy optimization, leaving the reliability and refinement of learned world models less explored. We propose a Bayesian offline MARL framework that infers latent global states from decentralized observation-action histories and iteratively refines a posterior world model using only offline data. Building on posterior information loss, we extend this criterion to the multi-agent setting as multi-agent posterior information loss (MPIL), which provides a reliability measure for world model refinement and offline evaluation. Theoretically, we establish an upper bound on policy regret that is monotonic in MPIL and further characterize its dependence on the size and state-joint-action coverage of the offline dataset. Experiments on partially observable multi-agent variants of D4RL MuJoCo tasks demonstrate competitive policy performance across diverse offline data conditions, while MPIL-guided refinement improves world model prediction and downstream policy performance. These results support a principled connection between world model reliability, offline data coverage, and policy performance in partially observable offline MARL.

View source

Similar papers

ICMAPE: In-Context Multiagent Pure Exploration

ICMAPE converts the fixed-confidence identification objective into a reward derived from inference confidence, so that standard reinforcement learning machinery can be applied to decentralized pure exploration.

Xin-Yi Hu, Alessio Russo, Aldo Pacchiano · 0 citations

MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimization

This work proposes Multi-Agent Flow-Pretrained Policy Optimization (MA-FPPO), which uses online fine-tuning to improve the cooperative behavior of models pretrained with flow matching through new interactions with the environment.

Guo-Wei Zou, Hao-Nan Chen, Hai-Tao Wang et al. · 0 citations

Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning

This paper proposes a model-based bootstrap framework for uncertainty quantification of OPE in finite-horizon, time-inhomogeneous Markov decision processes (MDPs), and establishes bootstrap distributional consistency, asymptotically valid confidence intervals, and consistent variance estimation for the target policy va...

Wei-Wei Wang, Yu-Qiang Li, Xian-Yi Wu et al. · 0 citations

MA-JEPA: Joint-Embedding World Models for Multi-Agent Reinforcement Learning

World models improve sample efficiency by training policies on imagined trajectories, but their usefulness depends on learning representations that capture the information needed for future control. We study whether self-supervised joint-embedding prediction (JEPA) can provide this learning signal for multi-agent reinf...

Brandon Kaplowitz, Osaze James Obahor, C. S. D. Witt · 0 citations
Oct 2026

Distributed variational bayesian meta-learning over multi-agent systems.

Learning new tasks from limited data inherently introduces uncertainty in meta-learning. While Bayesian meta-learning aims to enhance traditional algorithms by quantifying uncertainty in model parameters, existing methods are centralized and cannot handle decentralized data. In this paper, we present the Distributed Va...

Xian-Yang Zhang, Wei-Jie Kang, Ying Li et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.