Do LLMs Explain or Remember? Evaluating Language Model Explanations of Reinforcement Learning Failures

Nivetha Suntharamoorthy Jørn Eirik Betten Dennis Gross Eirik Valseth Helge Spieker

· 1 citation · 26 references

TL;DR

This study systematically investigates how prompt structure, model architecture, and environment representation affect the quality of Explainable RL (XRL) explanations, and conducts a multi-model evaluation using quantitative metrics to assess explanation coherence, helpfulness, and stability.

Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Shockingly Simple Self-retrospection Improves Agentic Models Without RL

People learn not only by repeating successful actions, but also by recounting and explaining their experiences, revising their understanding to guide future behavior. Can a language-model agent improve its future actions by training only on explanations of its own experience? We investigate this question by studying Re...

Jonathan Light, C. Cui, Jeonghye Kim et al. · 0 citations

#machine learning Preprint Aug 2026

Demystifying Reinforcement Learning Post-Training of Language Models

This work deconstructs the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface, and uses the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shape...

D. Clay, Saket Gollapudi, Sankar V Harilal et al. · 1 citation

#machine learning Preprint Aug 2026

Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents

EvalXRL is a benchmark in which a Large Language Model (LLM) coding agent uses different XRL methods to diagnose a held-out malfunction in an RL agent, and then repair it, and proposes the first head-to-head comparison of multiple XRL methods in closed-loop usage.

Ram Rachum, Yotam Amitai, Bálint Gyevnár et al. · 0 citations

#artificial intelligence Review Sep 2026

Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

This survey provides a systematic taxonomy of recent research on rollout efficiency for reasoning-oriented reinforcement learning, classifying existing approaches from both mechanism and bottleneck perspectives.

N. Gholipour, M. Assunção, Gursimran Singh et al. · 0 citations

#machine learning Preprint Sep 2026

RESCUE: Repairing Language Model Errors to Sparse Circuits via Reinforcement Learning

Large language models (LLMs) exhibit strong general capabilities that mechanistic interpretability has attributed to sparse computational circuits. However, existing circuit studies emphasize preserving functionality or explaining safety, leaving the mechanisms underlying failures across a broader range of tasks largel...

Chuan-Pu-Zou-Sheng-Li-Guo-Jing-Zhong-Gu-Yue-Shu-Ca Liu, Miao Yu, Yi-Kai Cai et al. · 0 citations

Delayed Convergence and Emergent CoT Reliance in RL-Tuned Language Models

This analysis reveals that intermediate log-probability is an unreliable indicator for reasoning capability; instead, reasoning performance results from a shift of internal confidence allocation where RL fine-tuning delays internal convergence, exhibiting prolonged mid-layer exploration before converging sharply at the...

Pablo Pérez-Lázaro, Rocío Aznar-Gimeno, F. J. Lacueva-Pérez et al. · 0 citations