Online Policy-Wise State Abstraction for Reinforcement Learning Agents Leveraging Free Energy.

Alireza Rashidi Laleh Majid Nili Ahmadabadi

Sep 2026 · IEEE Transactions on Neural Networks and Learning Systems · Vol PP · 0 citations

Medicine

TL;DR

This work proposes a dynamic, online, policy-wise state abstraction framework that constructs compact, task-relevant state representations (state abstraction candidates) during learning without prior knowledge, yielding lower regret and higher reward, particularly under limited learning budgets.

Abstract

Constructing compact, informative state representations from high-dimensional observations is a key challenge in reinforcement learning (RL). Directly mapping often introduces irrelevant or redundant dimensions in the state space. This causes the curse of dimensionality and increases regret, particularly during early learning. To address this challenge, we propose a dynamic, online, policy-wise state abstraction framework that constructs compact, task-relevant state representations (state abstraction candidates) during learning without prior knowledge. Our method uses parallel learners with a shared replay buffer to mitigate the potential adverse impact of abstraction on regret. One learner operates on the full state. Additional learners independently apply adaptive binary masking to dynamically select relevant state dimensions based on importance scores. At each decision step, a free energy-based metric is used to select the optimal learner for action selection. Experimental results demonstrate that our method achieves superior performance across diverse RL benchmarks, yielding lower regret and higher reward, particularly under limited learning budgets.

DOI PubMed Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments

Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions...

Gu-Heon Kang, Minhae Kwon · 0 citations

#artificial intelligence Sep 2026

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

Reward-Free Policy Optimization (RFPO), which repurposes a single calibrated, frozen critic as a rollout-level reward, a value baseline for generalized advantage estimation, and a success forecaster for unfinished prefixes is introduced.

Hong-Yang Li, Xiao Li, Caesar Wu et al. · arXiv.org · 0 citations

Review Open access Oct 2026

Reinforcement Learning: A Comprehensive Review of Foundations, Algorithms, and Open Challenges

Reinforcement learning (RL) is a branch of machine learning that studies how an agent should act within an environment to maximize cumulative reward. Supervised learning fits a function to labeled examples; RL instead learns through the consequences of trial-and-error interaction. It also differs from unsupervised lear...

Mohammad Hatim Eliwi, Waleed A. Hammod, O. A. Hammood et al. · EDRAAK · 0 citations

#artificial intelligence Preprint Sep 2026

Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching

Inverse Reinforcement Learning (IRL) aims to recover a reward function that explains expert demonstrations. Existing IRL methods typically rely on a bi-level optimization procedure that alternates between reward learning and policy optimization, leading to substantial computational burden and training instability. In t...

Yang Chen, Yi-Tan Zhang, Michael J. Witbrock et al. · 0 citations

#machine learning Preprint Sep 2026

Self-Confirming Superposition Traps in Reinforcement Learning

It is shown that this loop can sustain a lower-return policy even when representation fitting is globally optimal on data selected by the agent, which then uses the resulting returns to guide its next choices.

Dai Shi, Andi Han, Feng Chen et al. · 0 citations

#natural language process... Preprint Sep 2026

Expert-Space Exploration in MoE Reinforcement Learning

ESRL is introduced, an architecture-aware framework that explicitly explores the expert-routing space of MoE models, and preserves high-confidence experts as anchors, and restricts stochastic routing to a plausible candidate pool, thereby retaining reliable computation paths.

Hong-Yi He, Zheng-Wen Lin, Xiao Liu et al. · 0 citations