Skip to content

Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning

Sep 2026 · 0 citations · 48 references
Computer Science

TL;DR

Entropic Advantage Policy Optimization (EAPO) is introduced, an entropy-guided credit assignment method that treats success and failure asymmetrically and promotes more effective exploration, broadening problem coverage and generating more diverse candidate answers.

Abstract

Reinforcement learning with verifiable rewards (RLVR) enhances reasoning in large language models (LLMs) through outcome-level feedback, yet recent approaches to finer-grained credit assignment often require auxiliary models, additional sampling, or privileged information. Although policy entropy provides a readily available signal, prioritizing uncertain positions under both reinforcement and penalization concentrates penalties where failed responses still retain alternatives for recovery, which can suppress opportunities for exploration. To address this, we introduce Entropic Advantage Policy Optimization (EAPO), an entropy-guided credit assignment method that treats success and failure asymmetrically. Specifically, motivated by the observation that success under uncertainty is less repeatable while confident failures tend to recur, EAPO couples normalized policy entropy with the sign of the response advantage to reinforce surprising success and correct repeated failure. It assigns stronger reinforcement to high-entropy decisions in successful responses and stronger penalties to low-entropy decisions in failed responses, while attenuating penalties at uncertain positions to preserve opportunities for recovery. By redistributing the response advantage across tokens, EAPO derives token-level credit directly from existing rollout signals without additional supervision. We validate EAPO on a range of reasoning tasks across both base and reasoning backbones, demonstrating that it achieves the best overall performance. We further show that EAPO promotes more effective exploration, broadening problem coverage and generating more diverse candidate answers.

View source

Similar papers

Semifactual Credit-Augmented Policy Optimization

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its...

Jun-Shu Pan, Zhi-Zhang Fu, Shu-Lin Huang et al. · 0 citations

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

ProVer, a framework that targets potentially pivotal decisions for fine-grained credit assignment in agentic reinforcement learning, is introduced, highlighting the effectiveness and efficiency of selectively targeting pivotal decisions for fine-grained credit assignment in agentic reinforcement learning.

Dongwon Jung, H. Ramesh, Yi-Fan Wang et al. · 0 citations
Conference Aug 2026

Post-Training for Reasoning LLMs with Reinforcement Learning: A Stability–Efficiency Perspective

Reinforcement Learning (RL)-based post-training has become a key mechanism for improving reasoning Large Language Models (LLMs), especially when automated verifiers enable Reinforcement Learning with Verifiable Reward (RLVR). Recent methods, however, are increasingly modular: many apparent algorithmic advances recombin...

Liu Yang, Han Zhu, Zheng-Yang Zhong et al. · 2 citations
Preprint Aug 2026

Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals

Contrastive reinforcement learning (CRL) scales effectively in goal-conditioned tasks by casting policy learning into a self-supervised contrastive objective. However, in a failure-terminated Markov decision process, established CRL considers pre-failure future goals only when constructing positive samples, without acc...

Guopeng Li, Yingpeng Duan, Yiru Jiao et al. · 0 citations

UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning

UniOPSD (Unified On-Policy Self-Distillation) is introduced, which unifies these feedback sources through adaptive local credit arbitration and constructs comparable credit estimates from environmental returns and successful-peer hindsight at shared interaction anchors.

Zeng-Huang Fu, Zhao-Yang Li, Qiu-Yuan Ai et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.