This work formalizes LRM safety objectives as requiring non-malicious reasoning and answers, safety consistency, and explicit intent identification, and introduces a reasoning-preserving gradient projection mechanism that analytically resolves conflicts between safety and reasoning gradients, effectively eliminating th...
Yi-Chen Sun, Qi-Rui Liu, Yan Wang et al.· 0 citations
Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We in...
Outcome-reward reinforcement learning such as GRPO gives every token of a search-agent rollout the same advantage, although the rollout exercises two capabilities: finding evidence and using it. We show that accuracy improves without grounding: Search-R1 raises accuracy on all seven QA benchmarks, yet on Bamboogle the...
It is found that, for a substantial fraction of incorrect joint predictions, the question-relevant unimodal branch can still recover the correct answer, suggesting that task-relevant evidence can remain recoverable from the relevant modality even when it is not reflected in the joint prediction.
Yu Mi, Li-Cheng Pan, Jia-Yi Zhu et al.· 0 citations
DeShortcut-Align is proposed, a shortcut-decoupling alignment framework that reduces dependence on superficial cues that significantly improves robustness against template-stripping bypass attacks, substantially reduces over-refusal, and better preserves general-purpose reasoning capabilities, thereby mitigating the al...
Qi-Rui Liu, Yi-Chen Sun, Yan Wang et al.· arXiv.org· 2 citations
Efficient On-Policy Self-Distilled Safety Alignment (EOPSA), which concentrates computational and gradient budgets exclusively on reliably supervised, safety-critical tokens, is proposed, consistently outperforming full-token distillation baselines in both safety compliance and reasoning retention.
Qi-Rui Liu, Yi-Chen Sun, Yan Wang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.