Sep 2026· IEEE transactions on multimedia· Vol 28, pp. 14920-14933· 1 citation· 93 references
Computer Science
TL;DR
This paper proposes a novel phase-aware adversarial attack framework that explicitly restricts adversarial perturbations to structure-relevant phase regions to suppress the MLLM' focus for effective and imperceptible attacks.
Abstract
Multimodal large language models (MLLMs) have extended the capability of large language models (LLMs) to process more contextual multimodal information, showing remarkable progress in diverse realistic multimodal applications. Despite their strong perception and reasoning abilities, recent studies reveal that MLLMs remain highly vulnerable to adversarial inputs, especially those targeting visual components. However, existing attacks mainly focus on global perturbations, lacking an understanding of how MLLMs internally interpret visual structures. In this paper, we make the attempt to investigate the intrinsic focus of MLLMs in the frequency domain and discover that their predictions are particularly sensitive to phase information, which encodes essential structural and semantic cues. Based on this observation, we propose a novel phase-aware adversarial attack framework that explicitly restricts adversarial perturbations to structure-relevant phase regions to suppress the MLLMs' focus for effective and imperceptible attacks. To further amplify the structural influence, we also introduce an auxiliary adversarial prompt learning module to guide multimodal misalignment around phase-sensitive regions, misleading the MLLM's attention toward targeted structural patterns. Extensive experiments on multiple representative MLLM models and datasets demonstrate the superior effectiveness of our method compared to existing attacks.
Although Large Vision-Language Models (LVLMs) have shown great capabilities across various downstream tasks, they are proven to be vulnerable to adversarial examples, thus resulting in critical safety issues. Existing adversarial attacks against LVLMs predominantly rely on uniform, image-wise perturbations, while large...
Dai-Zong Liu, Tian-Yao Luo, Junhao Dong et al.· IEEE Transactions on Pattern...· 0 citations
Multimodal Large Language Models (MLLMs) can memorize and reproduce sensitive information from their training data, posing significant privacy risks. Their ability to encode knowledge across both textual and visual modalities further complicates the selective removal of such private information. Many existing methods g...
Lei-Jiang Gu, Zhen Zeng, Feng Li et al.· 0 citations
The proposed Multi-modal Text Distribution-Aware Deep Adversarial Prompt Tuning (MTDDAPT), which builds on multi-modal adversarial prompt coupling with frozen encoders and further introduces explicit distribution-aware regularization over class-level textual embeddings, is proposed.
Chao Li, Zhen Tian, Cai-Chang Ding et al.· Computers, Materials & C...· 0 citations
This work proposes Saliency-Driven Perceptual Realignment (SDPR), a training-free framework that mitigates the degradation of visual awareness throughout inference and is robust against hallucinations due to its holistic alignment of visual awareness across the entire generative trajectory.
Peng-Xu Chen, Yao Zhu, Guang-Ming Zhu et al.· Proceedings of the 34th ACM...· 0 citations
Visual-Language Pre-training (VLP) models have achieved remarkable success in various practical applications, yet they remain vulnerable to adversarial attacks that can mislead their predictions. Adversarial attacks are valuable for revealing the vulnerabilities and robustness issues of VLP models. However, existing st...
Despite the remarkable capabilities of Large Vision-Language Models (LVLMs), they remain vulnerable to jailbreak attacks. The integration of visual modalities further expands the attack surface due to its high-dimensional nature. To ensure the secure development of LVLMs, it is essential to develop more potent jailbrea...
Tao Huang, Rui Wang, Xiao-Fei Liu et al.· Proceedings of the 34th ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.