Reinforcement Learning with Scenario-Context Rollout in Portfolio Management

Abstract

When economic structures and market dynamics shift, classic portfolio rebalancing algorithms often suffer from unstable and degraded performance. To improve the return and robustness of portfolio management, we explore reinforcement learning (RL) and propose Scenario-Context Rollout (SCR), a macroeconomics-guided feedback mechanism to produce a distribution of next-day joint returns under potential economic shocks. However, doing so faces new challenges, as history will never tell what would have happened differently. As a result, incorporating scenario-based rewards from rollouts introduces a reward-transition mismatch in temporal-difference (TD) learning, destabilizing RL critic training. We theoretically analyze this problem and show that combining scenario-scored rewards with tape-realized transitions induces a hybrid fixed point. Guided by this analysis, we construct a counterfactual next state using the SCR continuations and augment the critic agent's bootstrap target. Doing so stabilizes the learning and provides a viable bias-variance tradeoff. In out-of-sample evaluations across 31 distinct universes of U.S. equity and ETF portfolios, our method improves Sharpe ratio by up to 76% and reduces maximum drawdown by up to 53% compared with classic and RL-based portfolio rebalancing baselines.