TL;DR
This work introduces Epistemic Regret Minimization (ERM), a frame work that critiques the causal structure of a model’s reasoning trace rather than its answer, outperforming outcome-only baselines 38-fold.
ACA-RL supports a new mission for NLP evaluation: measuring whether models can recognize when a task is underdetermined and handle uncertainty, not only whether they can answer fully specified questions.
Results provide evidence that access to a scalable, objective training signal can be a more important constraint than model scale alone, and can make otherwise ambiguous diagnostic reasoning tasks amenable to scalable reinforcement learning.
This work trains language models with GRPO on multiple-choice math problems where the correct answer is always option A, then evaluates on an unseen test set with unbiased answer positions to find reasoning-answer decoupling, which separates capability loss from a learned, transferable shortcut.
The mechanism is a halt vector: a difference-of-means direction at layer 18 of this model whose steering strength controls how long it thinks, while a replicated value axis does nothing, and what works is reconstructing the whole steered activation with those dimensions pinned to their natural values.
Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock ov...
A formal analysis showing that joint optimization of the two objectives induces gradient conflict in early training, motivating the sequential design of ERR+, a two-phase RLVR framework grounded in this observation.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
