The results validate that SPRL enables the development of trustworthy autonomous systems, paving the way for the practical deployment of RL in real-world supply chain operations.
Abstract
Reinforcement Learning (RL) offers significant potential for solving complex, sequential decision-making problems in supply chain management, such as dynamic inventory control. However, the adoption of state-of-the-art deep RL methods is severely hampered by their "black box" nature, as the resulting policies are notoriously opaque and difficult to trust in high-stakes industrial environments. To address this critical interpretability-performance gap, we introduce Structured Policy Reinforcement Learning (SPRL), a novel framework that embeds transparency directly into the agent's learning process. SPRL uses a decoupled architecture to distill value estimates from a powerful Q-learning agent into an inherently interpretable Decision Tree (DT). Crucially, the entire process is governed by a hard operational constraint (Inventory Position Cap, [Formula: see text]), which acts as an operational guardrail to stabilize training and enforce lean policies. We demonstrate that the optimized SPRL-DT policy achieves highly efficient combined performance, realizing a mean total cost of [Formula: see text] (a 45% reduction compared to the constrained DQN baseline) while maintaining an excellent service level of over 91%. This performance is achieved with a low complexity structure ([Formula: see text] nodes), offering a highly transparent and verifiable solution. Our results validate that SPRL enables the development of trustworthy autonomous systems, paving the way for the practical deployment of RL in real-world supply chain operations.
This work reviews pre-2019 XRL approaches, categorizing them into policy explanation, reward decomposition, model transparency, and post-hoc interpretability methods, and proposes a framework that combines interpretable policies, surrogate models, attention mechanisms, and visualization techniques to enhance transparency without significantly reducing performance.
Michael Anderson, David Thompson· International Journal of Art...· 0 citations
Comparative control theory analysis reveals a fundamental trade-off: transitioning from continuous to discrete rule-based control induces high-frequency Bang-Bang actuation and a stable bimodal limit cycle.
Decision Transformers (DTs) have emerged as a powerful paradigm for sequential decision-making in offline learning, yet they face two intrinsic deficiencies: the challenge of specifying optimal Return-to-Go (RTG) targets and a fundamental lack of exploration beyond the offline dataset. While prior studies have attempted to address these issues separately, their solutions introduce new problems. In this paper, we introduce LOGIC (Learning Optimal Goals with an Integrated Critic), a holistic framework that resolves these dual challenges in a unified manner. LOGIC transforms the exploration paradigm: instead of exploring at the action level, it employs an integrated critic to guide a generator in discovering optimal, high-value goals (RTGs). This strategic shift enables robust exploration beyond the dataset's frontier while avoiding the fragility of complex multi-loss objectives. Simultaneously, a Backward Consistency Refinement (BCR) module, applied during both training and inference, ensures that all generated plans remain mathematically and semantically valid. Extensive experiments on diverse auto-bidding scenes demonstrate that LOGIC significantly outperforms state-of-the-art baselines, achieving superior cumulative returns and model robustness, thereby providing a principled and unified solution to goal-oriented offline learning.
Shihao Shu, Rujie Zhong, Hao Wang et al.· Proceedings of the 32nd ACM...· 0 citations
This work introduces SPOT (Sampling Policy Observation Tree), a novel model-agnostic, sampling-based framework for interpreting DRL policies, and demonstrates how its tree-based representation can be used to inspect policy preferences, compare alternative future trajectories, and reveal downstream behaviors that are not visible through single-timestep feature-attribution methods.
This work proposes ARMOR (Anchor Rollout and Mixed Optimization for RL), a framework that shifts the paradigm from passive penalty to active sample stabilization, enabling sustained performance improvements over extended training horizons.
Kexin Huang, Junkang Wu, Jinda Lu et al.· 0 citations
Outcome-based reinforcement learning provides verified feedback for language-model agents, but assigns trajectory-level advantage uniformly to all decisions, yielding coarse credit over long-horizon interactions. On-policy self-distillation offers finer supervision by re-evaluating sampled behavior with privileged information (PI) available only during training. However, fine-grained supervision is not necessarily fine-grained credit: PI-induced likelihood changes describe how additional information alters policy preference, but do not directly determine how an executable action should inherit the verified task outcome. This creates a supervision-credit gap. Privileged signals may be irrelevant to the current interaction state, operate at a token granularity misaligned with executable decisions, and lack the outcome semantics required for reinforcement. We introduce TASPO, which converts privileged supervision into outcome-grounded action credit. TASPO constructs decision-applicable PI from verified successful experience, aggregates PI-induced likelihood shifts at the executable-action level, and converts relative action support into positive, bounded, mean-preserving weights on the original trajectory advantage. Thus, the verified outcome determines the update direction and average scale, while PI only redistributes credit across actions. Across three agentic benchmarks, TASPO improves over GRPO by 10.6\% and generalizes better to unseen tasks. Further analysis indicates that TASPO reduces supervision mismatch and that action-level assignment stabilizes the policy optimization process. These findings offer the community another interesting perspective.
Jingxiao Yang, Wangjie Gan, Ying-Xuan Zhuang et al.· 0 citations