This work employs Inductive Logic Programming (ILP) to extract symbolic representations of RL policies and define a novel set of explainability metrics, including activation rate, feature coverage, syntactic distance and semantic distance, which provide crucial insights for the transfer and generalization of action-specific policies.
Abstract
Explainable Reinforcement Learning (XRL) seeks to make Reinforcement Learning (RL) policies more transparent and interpretable, a key requirement in safety-critical and human-centric scenarios. However, it is mostly based on user studies, thus targeting the needs of a specific audience and lacking shared evaluation metrics. On the other hand, logic-based approaches within eXplainable Artificial Intelligence (XAI) provide compact, human-readable abstractions of decision-making. However, the systematic quantification of the explainability degree of logical representations remains an open problem. This work aims to advance the state of the art in XRL by introducing objective and planning-oriented metrics for policy explainability in RL settings. At the same time, it contributes to the field of logic for XAI by providing a principled way to quantify the explainability of logical rules, moving beyond common-sense assessments and simple propositional fragments. We employ Inductive Logic Programming (ILP) to extract symbolic representations of RL policies and define a novel set of explainability metrics, including activation rate, feature coverage, syntactic distance and semantic distance. These metrics quantify alignment between symbolic rules and agent behavior, the role of features in decision-making, and the evolution of policies during training and across agents in single and multi-agent RL. Experiments across different RL domains show that the proposed metrics highlight action-specific learning dynamics beyond global return, provide fine-grained insights into domain features beyond classical approaches for global feature importance estimation, and uncover coordination, specialization, and adaptation patterns in MARL. Moreover, they provide crucial insights for the transfer and generalization of action-specific policies.
This work reviews pre-2019 XRL approaches, categorizing them into policy explanation, reward decomposition, model transparency, and post-hoc interpretability methods, and proposes a framework that combines interpretable policies, surrogate models, attention mechanisms, and visualization techniques to enhance transparency without significantly reducing performance.
Michael Anderson, David Thompson· International Journal of Art...· 0 citations
Reinforcement Learning (RL) has been extensively used in autonomous and self-* systems, but RL policies, especially deep RL ones relying on neural networks, lack transparency and are difficult to understand. This can lead to diminished user trust, and makes for a more challenging verification of systems. To address this challenge, this paper introduces Explanations using Alternative Realities for Reinforcement Learning (EARL), a Python library to produce counterfactual explanations in RL settings. This library allows the user to produce explanations by exploring What-if scenarios to clarify agent behavior by comparing possible outcomes. Counterfactual explanations have been shown to be intuitive and user-friendly in psychology research, but have only recently been explored in RL, with existing implementations usually limited to toy examples and benchmarks. EARL supports counterfactual explanation generation in realistic RL-based self-adaptive systems. To demonstrate its applicability, we demonstrate its use in a simulation of CitiBikes, a self-adaptive bike-sharing system, and we provide evaluations showing how it performs in real applications.
Jasmina Gajcin, Juan C. Rosero, Ivana Dusparic· 0 citations
This paper attempts to outline the evolution of LLMs from classic methods of supervised finetuning models with static human-annotated datasets, to a more dynamic and evolutionary reinforcement learning based autonomous models. Along with the systematic evolution of reasoning capabilities, this is one of the most prominent focal points in the field of AI. This paper attempt to outline the most current advancements in reasoning and reinforcement learning, and classify the occurrences into the applicable areas of criteria, such as: choice of architecture, choice of reward assignments, and choice of evaluation metrics. This paper explore reasoning improvements as a result of self-reflective and exploratory processes within the bounds of the RL with Verifiable Rewards (RLVR) framework. By systematically studying the aforementioned criteria across multiple works and the respective variations in algorithmic efficiency, control of reward signal bias, and performance metrics, this paper want to outline the positive role of reinforcement learning in fostering autonomous self-correction in models and complex thought chain processes. The aim of this paper is to describe the potential autonomous advancements the next generations of large language models may evolve and want to offer some suggestions as a theoretical and a technical framework.
Mengbo Song· Mathematical Modeling and Al...· 0 citations
This thesis proposes a unified two-axis framework that organizes SFT and RL methods along a data axis (off-policy to on-policy) and a loss function axis (positive-only to positive-plus-negative to GRPO) and enables controlled ablations of individual components.
G. Kim, Chair Chenyan Xiong, Aditi Raghunathan· 0 citations
In recent years, there have been several developments combining reinforcement learning (RL) with techniques from theoretical computer science fields such as logic and formal methods. The main goal of these works was to improve training speed and quality, and in some cases also enforce safety constraints. In this dissertation, we present several works that followed this research line. First, we explore research directions concerning reward machines (RMs), an approach proposed to improve training speed and train agents in achieving tasks that require temporally extended behaviours. Given an abstraction of the environment in which the agent acts, we show how we can generate a reward machine from the set of all plans to achieve the task in the abstraction. As the plans come from an abstraction of the environment, the agent still needs to learn how to enact them in order to achieve the task, which is done via RL. Then, we synthesise reward machines in a cooperative multi-agent scenario by using Alternating-time Temporal Logic (ATL) formulas encoding coalition tasks. By model checking the ATL formula, we can obtain a strategy (if there is any) for the coalition to achieve the task, which is then translated to a RM and used to train the agents. We then present an extension of reward machines that endows them with a pushdown stack, obtaining a "pushdown reward machine" (pdRM). As pdRMs are based on pushdown automata, they can encode a strictly larger set of tasks compared to standard RMs, while still enabling more efficient learning compared to other approaches. Finally, we present a work in safe RL, where agents must also respect safety constraints. We present how to enforce safety constraints using pure-past linear-time temporal logic (PPLTL). Each action is associated to a PPLTL formula, and by evaluating the formulas at each timestep we determine which actions the agent can to perform, guaranteeing constraint satisfaction.