Reinforcement Learning for the Working Memory Controller
Abstract
Agentic systems built on large language models rely on a bounded-capacity working memory (WM) that holds the small set of items recent observations, active facts, in-progress subgoals immediately available for reasoning, while less pressing information is evicted or consolidated to a long-term store. Existing WM controllers rely on fixed heuristics such as recency (LRU), first-in-first-out ordering, or hand-tuned consolidation scores, all of which optimize a proxy signal rather than the downstream task objective the memory ultimately serves. This paper proposes WMC-RL, a working-memory controller trained with reinforcement learning that formulates admission, eviction, and consolidation as actions in a Markov decision process and learns a policy, via Proximal Policy Optimization, that directly optimizes a composite reward combining sparse task success with dense penalties for retrieval cost, memory churn, and unnecessary consolidation. WMC-RL is evaluated on LongHorizon-30, a 30-task benchmark of long-horizon agent episodes spanning multi-session research assistance, iterative code debugging, and long-document question answering, against random, FIFO, LRU, and a hand-tuned heuristic consolidation-score baseline. WMC-RL achieves a task success rate of 86.1%, exceeding the strongest heuristic baseline by 11.8 percentage points, reduces eviction regret by 53.2%, and improves consolidation precision to 79.8%. Complexity analysis, an ablation study, a sensitivity analysis over the reward-weighting coefficient, and a working-memory-capacity scalability study are reported, together with a qualitative case study of a learned consolidation decision and an explicit discussion of threats to validity.