Skip to content
Book Open access

Thinking Dangerously: How Reasoning Training Dilutes Safety in Language Models

Nov 2026 · Proceedings of the 2026 ACM SIGSAC Conference on Computer and Communications Security · 0 citations · 11 references

Abstract

Ask a reasoning model a harmful question and it refuses; ask it again after a long, entirely benign reasoning chain and it answers. Across six open-weight models, average refusal falls from 94.6% to 12.7% behind an 8,192-token benign prefix. We name this failure Safety Dilution and make it measurable: refusal decays smoothly with length, so each model has an L50, the prefix length at which refusal falls below one half, within a few thousand tokens, and the form-free Area Under the Bypass Curve (AUBC) makes the erosion comparable across models. In one matched-parameter, same-pretraining-base family, reasoning training doubles AUBC, though the effect also appears in models without it. Length-monotone safety is one of three properties a chain is assumed to have; the other two break as readily. Premise Corruption plants a false premise that the model's own inference propagates through the chain, largely evading prompt-injection defenses. Reasoning Chain Injection forges steps that match the model's own continuation distribution, blunting coherence checks wherever reasoning tokens are exposed. ChainGuard checks all three properties at runtime without retraining, cutting attack success by 56.3 percentage points on average; its truncation-only default recovers over half of that at 6% overhead. The gap does not close, but an adaptive adversary pays roughly two orders of magnitude more compute per success. Safety that holds for a sentence but dissolves over a chain is not safety: alignment has to hold over the whole computation, not just its opening tokens.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.