The Informationally Integrated Alignment Hypothesis: Integrated Information Aligns with and Predicts Final Reward in Reinforcement Learning Agents

Abstract

A hallmark of life on Earth is the ability of agents to exert causal power and be drivers of subsequent events. This is key to cognition at all scales. Integrated information, measuring the degree to which an agent exerts unique predictive power on its future, is one consequence of causal power. Indeed, recent discoveries have shown that biological agents, even minimal ones, increase their integrated information after learning new memories. However, there is a major knowledge gap regarding how informationally integrated artificial agents are. We focused on Reinforcement Learning (RL) of neural-network agents across a range of environmental conditions, encompassing different algorithms, agent architectures, and six environments arranged along a complexity spectrum. For consistency, we computed the integrated information of their latent-space representations over their lifetimes. We used the recently proposed ΦID to estimate integrated information and tested how it related to learning performance. Our results suggested a Causally Emergent Alignment Hypothesis: successful agents exhibited integrated information that was consistently predictive of final reward early in training and whose representational dynamics aligned with reward improvement in most tasks. This idea suggests that integrated information may be a previously undisclosed axis of reorganization of neural representations in RL agents, with the potential to establish causal relationships and interventions that will lead to better RL agents. Our work also highlights the alignment between integrated information and learning as another way biological and artificial creatures compare. Data/Code available at: https://github.com/pigozzif/PhiRL