Skip to content
Book Open access

DIRE: Dual-grained Internal Refusal Estimation for VLM Jailbreak Detection

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 12 references

Abstract

While Vision-Language Models (VLMs) excel at multimodal tasks, they are increasingly vulnerable to sophisticated multimodal jailbreak attacks. Conventional defenses typically introduce significant training overhead, risk degrading model capabilities, or cause severe inference latency via external modules. Additionally, current internal state detection methods rely on coarse-grained feature analysis, often failing to capture the micro-level conflicts generated by jailbreak attacks. To overcome these challenges, we analyze multimodal jailbreaks through mechanistic interpretability. We demonstrate that successful attacks typically do not eradicate intrinsic safety knowledge but instead cause a widespread attenuation of refusal signals. Even when global safety alignment is bypassed, a small, robust subset of Attention and FFN components continues to exhibit directional resistance against malicious queries. Building upon this empirical finding, we introduce Dual-grained Internal Refusal Estimation (DIRE), a training-free detection framework. Unlike prior internal monitors that are either macro-only or attention-only, DIRE jointly probes resilient Attention routing and FFN semantic memory within effective semantic layers to detect multimodal jailbreaks in a training-free manner. Comprehensive evaluations on mainstream VLMs and diverse benchmarks demonstrate that DIRE achieves competitive and strong performance, with an average AUROC of 0.9800. For better reproducibility, the anonymized code is available at: https://github.com/xiyuzzz-njust/DIRE.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.