Skip to content

Learning to Decide, Not to Reason: Parameter-Efficient Decision Operators via Low-Rank Activation Steering

Oct 2026 · 2 citations · 46 references
Computer Science

TL;DR

DecSteer, a System-1 decision operator trained by behavior cloning that lowers this cost of injecting skills into a frozen language model by roughly two orders of magnitude, is introduced.

Abstract

Injecting skills into a frozen language model currently costs a million parameters and a reinforcement-learning pipeline. We introduce DecSteer, a System-1 decision operator trained by behavior cloning that lowers this cost by roughly two orders of magnitude. The default operator uses 330K parameters to match a 1.33M-parameter operator trained with reinforcement learning, exceeds or achieve comparable performance, while collapsing 3,685-token deliberation into a 6-token decision with no loss in accuracy. A rank-4 variant with 23K parameters, 1/58 of the strongest published skill operator, suffices for SearchQA and near-suffices for LiveMath, where higher rank still helps; the same recipe transfers across five tasks and three backbones, with out-of-distribution gains persisting on LiveMath problems released months after training. The gap to prior work is trainability, and it is set jointly by initialization and architecture. The initialization of prior operators zeroes the gradient of both large factor matrices at the first optimization step, whereas our zero-initialized output projection inside a shared low-rank backbone receives a gradient immediately, which a gradient-flow probe confirms directly. The gain isn't chain-of-thought compression. 23 of 57 LiveMath points beat the base model's best-of-8 sampling, and a logit-lens probe shows the operator amplifies the answer along the model's existing late-layer pathway, not writing it earlier. Gains track the base model's headroom across 13 base-task pairs, and skills compose as approximately linear operators that can be added, interpolated, and hot-swapped at inference time.

View source

Similar papers

#small language model Book Open access Nov 2026

POLAR: Policy Optimization through Larger Reasoners

POLAR is presented, a difficulty-aware training framework that augments on-policy RL with conditional guidance from a larger reasoner that can turn otherwise wasted all-zero groups into targeted learning opportunities while keeping policy exploration on prompts the student can handle.

Ye Yuan, Zhi-Ping Xiao, Ming Zhang · 0 citations

Frontier Learning: Training LLM Reasoners at the Edge of Capability

Across several reasoning tasks and model families, the proposed frontier learning approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.

Robin Faro, S. Ramesh, Ilija Bogunovic et al. · 0 citations

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

A label-free bias-only TTRL, which uses majority-vote pseudolabels as rewards and optimizes only ~100K bias parameters while keeping the pretrained backbone frozen, demonstrating that substantial test-time adaptation can emerge from optimizing a tiny bias-only subspace using entirely label-free rewards.

Naveen Vakada, Ming-Yuan Li, Shao-Xiong Ji · 1 citation · ⚡1
Review Open access Sep 2026

Reinforcement Learning Post-Training for Reasoning Large Language Models: Methods, Systems, and Evaluation

This survey develops a mechanism-driven framework for separating effects of reinforcement learning, and organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, den...

Liu Yang, Han Zhu, Zheng-Yang Zhong et al. · 1 citation
#machine learning Preprint Oct 2026

Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

A local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions is proved, offering a conditional explanation for the success of self-RL teachers in the authors' experiments.

Lei Zhao, Qi-Chao Zhao, Bo-Wen Zuo et al. · 0 citations
#artificial intelligence Preprint Oct 2026

When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding...

Ze-Hao Wang, Han-Zhu Chen, Run-Quan Gui et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.