2026· IEEE Transactions on Information Forensics and Security· Vol 21, pp. 7157-7172· 6 citations· 51 references
Computer Science
Abstract
As the capabilities of Vision Language Models (VLMs) continue to improve, they are increasingly targeted by jailbreak attacks. Existing defense methods face two major limitations: (1) they struggle to ensure safety without compromising the model’s utility; and (2) many defense mechanisms significantly reduce the model’s generation efficiency. To address these challenges, we propose SafeSteer, a lightweight inference-time steering framework that effectively defends against diverse jailbreak attacks without modifying model weights. At the core of SafeSteer is the innovative use of singular value decomposition (SVD) to purify a low-dimensional “safety subspace” from noisy activation differences. By projecting the raw steering vector into this subspace, SafeSteer isolates the core safety signal from noise, adaptively removing harmful influences while preserving the model’s ability to handle benign inputs. SafeSteer avoids iterative response generation and introduces only limited overhead compared with other single-pass activation-steering defenses. Extensive experiments show that SafeSteer reduces the attack success rate by over 60% while maintaining the model’s utility on benign tasks, without introducing significant inference latency. These results demonstrate that robust and practical jailbreak defense can be achieved through simple, efficient inference-time control.
Activation steering offers an inference-time defense for vision--language models (VLMs) by modifying intermediate representations without updating backbone parameters. However, protection on benchmark inputs may not persist across alternative expressions of the same harmful request. We investigate this gap using fixed...
Xin-Wei Zhang, Ao-Ting Hu, Hang-Cheng Liu et al.· 0 citations
Results reveal a systemic vulnerability of frontier VLMs to meta-adaptive jailbreaks to meta-adaptive jailbreaks and motivate defenses against meta-level adversaries.
Ben-Lei Cui, Sheng-Yuan Pang, Yu-Ke Wang et al.· 1 citation
AlcaTRAz (Anchored Tree-Rule defense Against jailbreaks), a prompt-level defense based on rule trees that operates exclusively on the input text and requires no modification or retraining of the target model, is proposed.
J. Res, Petr Kaska, Martin Perešíni et al.· 0 citations
A fine-tuning-stage defense that simultaneously hardens LLMs against both attack classes by redistributing safety signals across a broader set of neurons, and provides a formal guarantee that NeuronGuard strictly reduces the attack success rate (ASR) upper bound.
Anjun Gao, Yueyang Quan, Yu Xia et al.· 2 citations
The safety alignment of large language models (LLMs) faces persistent challenges from jailbreak attacks. While existing methods mostly leverage prompt engineering or adversarial optimization, we identify and formalize an underexplored semantic isomorphism vulnerability where harmful and safe scenarios share highly cons...
Fan Yang, Ke Wang, Wenzhou Dou et al.· IEEE Transactions on Audio,...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.