This work formalizes LRM safety objectives as requiring non-malicious reasoning and answers, safety consistency, and explicit intent identification, and introduces a reasoning-preserving gradient projection mechanism that analytically resolves conflicts between safety and reasoning gradients, effectively eliminating th...
Yi-Chen Sun, Qi-Rui Liu, Yan Wang et al.· 0 citations
Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We in...
Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a binary judgment from an LLM verifier. However, judgment certainty varies substantially across criteria. Existing methods uniformly aggregate these judgments into a final...
Li-Cheng Pan, Hao Yang, Hao-Xuan Li et al.· 0 citations
DeShortcut-Align is proposed, a shortcut-decoupling alignment framework that reduces dependence on superficial cues that significantly improves robustness against template-stripping bypass attacks, substantially reduces over-refusal, and better preserves general-purpose reasoning capabilities, thereby mitigating the al...
Qi-Rui Liu, Yi-Chen Sun, Yan Wang et al.· arXiv.org· 2 citations
Efficient On-Policy Self-Distilled Safety Alignment (EOPSA), which concentrates computational and gradient budgets exclusively on reliably supervised, safety-critical tokens, is proposed, consistently outperforming full-token distillation baselines in both safety compliance and reasoning retention.
Qi-Rui Liu, Yi-Chen Sun, Yan Wang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.