Skip to content
Preprint

Bridging Exploration and Loss of Plasticity in On-Policy Reinforcement Learning

Oct 2026 · 0 citations · 46 references
Computer Science

TL;DR

This work reports multiple metrics which characterize the effect that the policy standard deviation $\sigma$ has on LoP, and suggests related explanations for the effectiveness of widely adopted LoP mitigation techniques.

Abstract

Deep reinforcement learning (RL) algorithms have been observed to lose the ability to adaptively learn from new data, a phenomenon termed Loss of Plasticity (LoP). While many metrics have been shown to correlate with it, the causal mechanisms and the geometric conditions from which it can arise have not been clearly identified. We analyze LoP in Proximal Policy Optimization with isotropic Gaussian policies, the state-of-the-art formulation for many continuous action spaces in RL. We report multiple metrics which characterize the effect that the policy standard deviation $\sigma$ has on LoP, and we also suggest related explanations for the effectiveness of widely adopted LoP mitigation techniques. We observe a significant effect of $\sigma$ on the extent of LoP, and accompany this with both theoretical and experimental supporting arguments. Our experimental validation consists of LoP benchmark evaluations against standard baselines as well as real-world policy deployment on a 7-DoF robotic arm.

View source

Similar papers

Preprint Oct 2026

Towards Understanding the Impact of Plasticity Loss on Reinforcement Learning in Stochastic Environments

Deep reinforcement learning (RL) is a powerful framework for sequential decision-making. Despite many high-profile successes in recent years, deep RL is known to be brittle. An emerging line of research attributes this brittleness to a loss of plasticity in neural networks. However, plasticity loss has so far been unde...

Philipp Bordne, André Biedenkapp · 0 citations

Efficient Multi-Objective Reinforcement Learning with Expressive Function Approximators

Two tractable ways of modeling distributional returns in MORL are proposed and combined with recent advances in function approximators for reinforcement learning show improvements in function approximation that substantially improve the quality of the produced solution sets.

Adam Štafa, Petr Novotný · 0 citations
#machine learning Preprint Oct 2026

Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

A local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions is proved, offering a conditional explanation for the success of self-RL teachers in the authors' experiments.

Lei Zhao, Qi-Chao Zhao, Bo-Wen Zuo et al. · 0 citations
Review Open access Oct 2026

Reinforcement Learning: A Comprehensive Review of Foundations, Algorithms, and Open Challenges

Reinforcement learning (RL) is a branch of machine learning that studies how an agent should act within an environment to maximize cumulative reward. Supervised learning fits a function to labeled examples; RL instead learns through the consequences of trial-and-error interaction. It also differs from unsupervised lear...

Mohammad Hatim Eliwi, Waleed A. Hammod, O. A. Hammood et al. · 0 citations
#machine learning Preprint Oct 2026

Understanding Enrichment in Reinforcement Learning

When rewards are sparse, reinforcement learning with verifiable rewards (RLVR) often uses hints or intermediate guidance to generate more successful rollouts. This enrichment biases policy-gradient updates unless corrected via importance weights, but existing methods omit correction or truncate importance weights in or...

Jinwoo Kim, Shraddha Barke · 0 citations
#machine learning Preprint Oct 2026

Reinforcement Learning via Advantage Guidance

Reinforcement learning (RL) holds the promise of learning a policy that can outperform what is shown in the data. However, in practice, such systems suffer from instability and hyperparameter sensitivity. In contrast, supervised learning provides a stable and scalable training objective, but is fundamentally limited by...

Zhi-Yuan Zhou, Sergey Levine · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.