Learning Universal Semantic Simplex for Jailbreaking Large Vision-Language Models
Abstract
Despite the remarkable capabilities of Large Vision-Language Models (LVLMs), they remain vulnerable to jailbreak attacks. The integration of visual modalities further expands the attack surface due to its high-dimensional nature. To ensure the secure development of LVLMs, it is essential to develop more potent jailbreak attacks to probe their safety boundaries. This work focuses on universal attacks as they better reflect real-world scenarios. However, achieving robust universality remains challenging due to the tendency to overfit specific semantics within the training corpus. To bridge this gap and enhance generalization across diverse instructions, we propose Learning Universal Semantic Simplex (LUSS), a novel jailbreak method that shifts the optimization paradigm from point-wise perturbations to continuous adversarial simplices. By incorporating a sharpness-aware penalty and a simplex-based update strategy, LUSS captures a broader range of adversarial semantics within a convex hull to mitigate overfitting. To facilitate more rigorous evaluation, we introduce HarmInstBench, a manually curated benchmark with hierarchical annotations for diverse malicious scenarios. Extensive experiments on multiple LVLMs, including MiniGPT-4, LLaVA-v1.5, and Qwen2.5-VL, demonstrate that LUSS significantly outperforms state-of-the-art baselines, achieving up to a 60.8% relative improvement in Attack Success Rate (ASR), 31.8% in toxicity score, and a 44.7% gain in Malicious Intent Fulfillment Rate (MIFR). Warning: this paper contains harmful text!