Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning

Abstract

Sparse-reward reinforcement learning often fails because rollouts from the unassisted evaluation start rarely reach later task stages. Reverse curricula address this by starting some training rollouts from easier intermediate states, called scaffolds. The main choices are which intermediate starts to use (scaffold access) and how to distribute training across them (scaffold allocation). Most prior curricula pace removal on one shared schedule, which can fail when task instances, or contexts, learn at different rates. We introduce an online, learner-agnostic reset controller that gives every context its own curriculum. We use binary rollout success to decide whether to move a context's start forward or back. A stalled frontier also gets an occasional harder-start trial. A counting construction shows that synchronized global pacing can be insufficient when contexts need conflicting amounts of assisted practice. Across six navigation and manipulation settings, scaffold access improves learning and enables success in three where unassisted training fails within the reported budget. None of the global schedules we tested solved both groups in the constructed conflict. The per-context controller did. Group-level pacing works if the labels capture how contexts learn. Without those labels, per-context pacing still handles differences within a group. These results suggest pacing assistance separately for contexts that learn at different rates.