Skip to content

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

Sep 2026 · arXiv.org · Vol abs/2609.36254 · 0 citations · 76 references
Computer Science

TL;DR

This work introduces DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency, and proposes SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers.

Abstract

Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning before producing final answers. However, RL rewards are typically assigned based on final answers, providing little or no direct supervision over intermediate reasoning. This can lead to deceptive safety alignment, where the reasoning trace and final answer convey inconsistent safety signals. To systematically investigate this phenomenon, we introduce DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency. Across multiple LRMs and benchmarks, we find that deceptive safety alignment is pervasive under standard prompting conditions and is substantially amplified under prefilling attacks. We further provide a hidden representation analysis showing that models exhibit stronger safety discrimination at the final-answer stage than during intermediate reasoning. To close this gap, we propose SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers, encouraging early harmful intent recognition and enforcing reasoning-answer consistency. Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility. Code is available at https://github.com/xzhou98/SARA.

View source

Similar papers

Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Models

While Large Reasoning Models (LRMs) excel at complex reasoning, alignment through reinforcement learning often induces systemic overconfidence. In production environments, where logits may be unavailable, robust black-box uncertainty quantification (UQ) is essential for trustworthiness and safety. Focusing on question-...

Lucas Biechy, Cédric Eichler, Adrien Boiret et al. · 0 citations

First Token Matters: Understanding Safety Collapse in Large Reasoning Models

It is found that the refusal-related signal of LRMs drops sharply at the first generated token under harmful queries, which is associated with unsafe response generation, and this work proposes SafeToken, a lightweight inference-time intervention that injects a learned continuous safety anchor precisely at reasoning on...

Yi-Zheng Yang, Hai-Ning Yu, Yue-Chen Wang et al. · 0 citations
Aug 2026

Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty

A representation-space analysis of RIM and the Safety-Direction Penalty (SDP), which penalizes movement along a learned safety direction during reasoning fine-tuning, is provided, which restores safety while preserving benchmark reasoning performance.

Yipeng Zhao, Qi-Shun Yang, Shenzhe Zhu et al. · 0 citations
#artificial intelligence Conference Open access Sep 2026

On the Efficiency-Safety Dilemma in Large Reasoning Models

It is found that while efficiency methods seemingly reduce the success rate of jailbreak attacks, this improvement is often superficial and largely arises from degraded reasoning capabilities leading to"attempted but failed"malicious responses, rather than an increase in genuine alignment.

Yifei Yang, Zou-Ying Cao, Xingrui Wang et al. · 1 citation
Preprint Aug 2026

Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces

Reasoning Jury, a system that replaces the single judge with a jury of LLMs and a moderated consensus mechanism, is introduced to improve the fidelity of judgments for identifying reasoning defects and can be leveraged to understand failure modes of reasoning LLMs on benchmarks, which allows much deeper understanding o...

Congchao Wang, D. Singh, Qiaozi Gao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning

Distilling the reasoning capabilities of large language models (LLMs) into smaller students is a central challenge for efficient deployment. Current approaches face a fundamental tension: optimizing purely for verifiable task rewards (e.g., via GRPO) leads to reward hacking, where students arrive at correct final answe...

Matthieu Zimmer, Xiao-Tong Ji, Tu Nguyen et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.