DIRE: Dual-grained Internal Refusal Estimation for VLM Jailbreak Detection
Abstract
While Vision-Language Models (VLMs) excel at multimodal tasks, they are increasingly vulnerable to sophisticated multimodal jailbreak attacks. Conventional defenses typically introduce significant training overhead, risk degrading model capabilities, or cause severe inference latency via external modules. Additionally, current internal state detection methods rely on coarse-grained feature analysis, often failing to capture the micro-level conflicts generated by jailbreak attacks. To overcome these challenges, we analyze multimodal jailbreaks through mechanistic interpretability. We demonstrate that successful attacks typically do not eradicate intrinsic safety knowledge but instead cause a widespread attenuation of refusal signals. Even when global safety alignment is bypassed, a small, robust subset of Attention and FFN components continues to exhibit directional resistance against malicious queries. Building upon this empirical finding, we introduce Dual-grained Internal Refusal Estimation (DIRE), a training-free detection framework. Unlike prior internal monitors that are either macro-only or attention-only, DIRE jointly probes resilient Attention routing and FFN semantic memory within effective semantic layers to detect multimodal jailbreaks in a training-free manner. Comprehensive evaluations on mainstream VLMs and diverse benchmarks demonstrate that DIRE achieves competitive and strong performance, with an average AUROC of 0.9800. For better reproducibility, the anonymized code is available at: https://github.com/xiyuzzz-njust/DIRE.