Skip to content

SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision Language Models

2026 · IEEE Transactions on Information Forensics and Security · Vol 21, pp. 7157-7172 · 6 citations · 51 references
Computer Science

Abstract

As the capabilities of Vision Language Models (VLMs) continue to improve, they are increasingly targeted by jailbreak attacks. Existing defense methods face two major limitations: (1) they struggle to ensure safety without compromising the model’s utility; and (2) many defense mechanisms significantly reduce the model’s generation efficiency. To address these challenges, we propose SafeSteer, a lightweight inference-time steering framework that effectively defends against diverse jailbreak attacks without modifying model weights. At the core of SafeSteer is the innovative use of singular value decomposition (SVD) to purify a low-dimensional “safety subspace” from noisy activation differences. By projecting the raw steering vector into this subspace, SafeSteer isolates the core safety signal from noise, adaptively removing harmful influences while preserving the model’s ability to handle benign inputs. SafeSteer avoids iterative response generation and introduces only limited overhead compared with other single-pass activation-steering defenses. Extensive experiments show that SafeSteer reduces the attack success rate by over 60% while maintaining the model’s utility on benign tasks, without introducing significant inference latency. These results demonstrate that robust and practical jailbreak defense can be achieved through simple, efficient inference-time control.

View source

Similar papers

Preprint Sep 2026

SteerProbe: Learning to Bypass Safety Steering in Vision-Language Models

Activation steering offers an inference-time defense for vision--language models (VLMs) by modifying intermediate representations without updating backbone parameters. However, protection on benchmark inputs may not persist across alternative expressions of the same harmful request. We investigate this gap using fixed...

Xin-Wei Zhang, Ao-Ting Hu, Hang-Cheng Liu et al. · 0 citations
Preprint Sep 2026

AlcaTRAz - Anchored Tree-Rule Defense Against Jailbreaks

AlcaTRAz (Anchored Tree-Rule defense Against jailbreaks), a prompt-level defense based on rule trees that operates exclusively on the input text and requires no modification or retraining of the target model, is proposed.

J. Res, Petr Kaska, Martin Perešíni et al. · 0 citations
Preprint Aug 2026

NeuronGuard: Robust LLM Safety Alignment via Ablation-Aware Safety Signal Redistribution

A fine-tuning-stage defense that simultaneously hardens LLMs against both attack classes by redistributing safety signals across a broader set of neurons, and provides a formal guarantee that NeuronGuard strictly reduces the attack success rate (ASR) upper bound.

Anjun Gao, Yueyang Quan, Yu Xia et al. · 2 citations

Jailbreaking Jailbreaks: A Proactive Defense for LLMs

P RO A CT represents an orthogonal defense strategy that serves as an additional guardrail to enhance LLM safety against the most effective attacks.

Wei-Liang Zhao, Daniel Ben-Levi, Jinjun Peng et al. · 1 citation
2026

Semantic Isomorphism Attacks and Defense Evaluation for Jailbreaking Large Language Models

The safety alignment of large language models (LLMs) faces persistent challenges from jailbreak attacks. While existing methods mostly leverage prompt engineering or adversarial optimization, we identify and formalize an underexplored semantic isomorphism vulnerability where harmful and safe scenarios share highly cons...

Fan Yang, Ke Wang, Wenzhou Dou et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.