Skip to content

Downstream-Aware Context Selection for Online In-Context Reinforcement Learning

Sep 2026 · 0 citations · 35 references
Computer Science

TL;DR

A bounded-history context-management framework that predicts the task-dependent downstream effect of removing historical interactions to guide history selection and determine a decision-dependent context budget is proposed.

Abstract

In-context reinforcement learning (ICRL) enables large language model agents to adapt to new environments using their interaction history without updating model parameters. However, repeatedly conditioning on growing histories can lead to substantial token cost. We propose a bounded-history context-management framework that predicts the task-dependent downstream effect of removing historical interactions to guide history selection and determine a decision-dependent context budget. Formally, our framework uses the full rolling history as a reference. The predictor evaluates removal effects, defines a deletion ordering, and applies a shared selection criterion to determine how much history to retain at each decision. We evaluate the method in closed-loop SUMO driving under held-out in-distribution, unseen-domain, and unseen-route settings, and in ScienceWorld under a continual ICRL protocol. Relative to a baseline using the full context, our method reduces total token usage by 25.7%, 25.8%, and 23.2% across the three driving settings while maintaining comparable closed-loop driving performance. In ScienceWorld, it reduces total token usage by 52.1% compared to full context and uses 30.2% and 37.8% fewer tokens than the Recent and Similarity baselines, respectively, while maintaining performance.

View source

Similar papers

#machine learning Preprint Oct 2026

Considering Context: When World Models Need Context Encoders

Methods for generalization in model-based reinforcement learning typically assume that an agent cannot recover the latent context governing the environment dynamics from its own experience, and therefore supplies it externally. We formalize and test this assumption with \emph{predictive sufficiency}, which quantifies w...

Oleg Smirnov, Sofiane Ennadir, John Pertoft et al. · 0 citations
#natural language process... Preprint Aug 2026

ContextPilot: Teaching Agents for Proactive Context Management via Fine-grained RL

An RL method tailored for context management is proposed, which uses context and entropy variation to identify critical editing decisions for branch sampling and estimates action-level advantages from all branched trajectories that pass through the corresponding context editing action.

Zhuo-Shi Pan, Qizhi Pei, Jun-Ru Lu et al. · 1 citation

TTSE: A Two-Track Online Self-Evolution Framework

This paper proposes TTSE (Two-Track Self-Evolution), a dual-track online self-evolution framework that separates evolving knowledge into FACT (environmental facts, whose reliability is continuously verified through interaction evidence) and TIP (task-conditioned implementation procedures), and broadly compatible with e...

Rui-Min Pei, Yong-Kang Wu, Shang-Yi Zheng et al. · 1 citation

Beyond Skill Evolution: Self-Evolving Context Management Policies for Long-Horizon Agent Harnesses

ContextEvo, a framework that learns a context policy from long-horizon trajectories and reconstructs the model-visible context at key decision points, identifies context-related failures, and applies targeted policy updates, is introduced.

Wei-Yuan Li, Jing-Heng Xu, Ai-Li Chen et al. · 0 citations
Review Open access Sep 2026

Reinforcement Learning Post-Training for Reasoning Large Language Models: Methods, Systems, and Evaluation

This survey develops a mechanism-driven framework for separating effects of reinforcement learning, and organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, den...

Liu Yang, Han Zhu, Zheng-Yang Zhong et al. · 1 citation

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.