DecSteer, a System-1 decision operator trained by behavior cloning that lowers this cost of injecting skills into a frozen language model by roughly two orders of magnitude, is introduced.
Abstract
Injecting skills into a frozen language model currently costs a million parameters and a reinforcement-learning pipeline. We introduce DecSteer, a System-1 decision operator trained by behavior cloning that lowers this cost by roughly two orders of magnitude. The default operator uses 330K parameters to match a 1.33M-parameter operator trained with reinforcement learning, exceeds or achieve comparable performance, while collapsing 3,685-token deliberation into a 6-token decision with no loss in accuracy. A rank-4 variant with 23K parameters, 1/58 of the strongest published skill operator, suffices for SearchQA and near-suffices for LiveMath, where higher rank still helps; the same recipe transfers across five tasks and three backbones, with out-of-distribution gains persisting on LiveMath problems released months after training. The gap to prior work is trainability, and it is set jointly by initialization and architecture. The initialization of prior operators zeroes the gradient of both large factor matrices at the first optimization step, whereas our zero-initialized output projection inside a shared low-rank backbone receives a gradient immediately, which a gradient-flow probe confirms directly. The gain isn't chain-of-thought compression. 23 of 57 LiveMath points beat the base model's best-of-8 sampling, and a logit-lens probe shows the operator amplifies the answer along the model's existing late-layer pathway, not writing it earlier. Gains track the base model's headroom across 13 base-task pairs, and skills compose as approximately linear operators that can be added, interpolated, and hot-swapped at inference time.
POLAR is presented, a difficulty-aware training framework that augments on-policy RL with conditional guidance from a larger reasoner that can turn otherwise wasted all-zero groups into targeted learning opportunities while keeping policy exploration on prompts the student can handle.
Ye Yuan, Zhi-Ping Xiao, Ming Zhang· Proceedings of the 35th ACM...· 0 citations
Across several reasoning tasks and model families, the proposed frontier learning approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.
Robin Faro, S. Ramesh, Ilija Bogunovic et al.· arXiv.org· 0 citations
A label-free bias-only TTRL, which uses majority-vote pseudolabels as rewards and optimizes only ~100K bias parameters while keeping the pretrained backbone frozen, demonstrating that substantial test-time adaptation can emerge from optimizing a tiny bias-only subspace using entirely label-free rewards.
This survey develops a mechanism-driven framework for separating effects of reinforcement learning, and organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, den...
Liu Yang, Han Zhu, Zheng-Yang Zhong et al.· Unmanned Systems· 1 citation
A local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions is proved, offering a conditional explanation for the success of self-RL teachers in the authors' experiments.
Lei Zhao, Qi-Chao Zhao, Bo-Wen Zuo et al.· 0 citations
Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding...
Ze-Hao Wang, Han-Zhu Chen, Run-Quan Gui et al.· 1 citation
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.