This work proposes Feature-Aware Token Attack (FATA), which couples attention suppression with cosine-based feature preservation on a fixed set of salient clean-image tokens and has the lowest measured detection rate among four attacks across three evaluated detectors at a 5% false-positive rate.
Abstract
Visual-token compression improves the efficiency of large vision-language models, but can expose failures that full-token evaluation misses. We study adversarial images that preserve full-token correctness yet induce errors after compression, even when both inference paths succeed on the clean image. Creating such failures is challenging because perturbing token importance can also damage the visual content needed for full-token inference. We propose Feature-Aware Token Attack (FATA), which couples attention suppression with cosine-based feature preservation on a fixed set of salient clean-image tokens. In the primary LLaVA-1.5-7B setting, FATA uses only vision-encoder gradients, without access to the deployed compressor, token budget, or downstream task. Across four visually dependent task subsets and four compressors under a controlled reconstruction protocol, FATA achieves SR = 96.3% full-token accuracy retention and CBR = 22.1% conditional blinding, compared with 89.8% and 15.7% for CAA. Ablations support the role of both objectives in balancing compressed-path failure against full-token preservation. FATA also has the lowest measured detection rate among four attacks across three evaluated detectors at a 5% false-positive rate. These findings motivate assessing adversarial robustness jointly across full-token and compressed inference.
Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression, demonstrating its practicality for resource-constrained LVLM deployment.
CatchUp is proposed, a training-free framework combining reversible visual-token routing with Local Transition Correction that achieves the best grounding performance across all eight splits and the highest average multimodal retention among evaluated efficient methods at 64-, 128-, and 192-token budgets.
Visual token compression reduces the inference cost of Large Vision-Language Models (LVLMs). However, aggregate robustness measures do not reveal whether a particular adversarial failure is induced by compression or inherited from the underlying model. We define a compression-specific failure (CSF) as an adversarial in...
Qian-Kun Li, Yuechen Zhang, Bo-Wen Chen et al.· 1 citation
Although Large Vision-Language Models (LVLMs) have shown great capabilities across various downstream tasks, they are proven to be vulnerable to adversarial examples, thus resulting in critical safety issues. Existing adversarial attacks against LVLMs predominantly rely on uniform, image-wise perturbations, while large...
Dai-Zong Liu, Tian-Yao Luo, Junhao Dong et al.· IEEE Transactions on Pattern...· 0 citations
Foveated Compression is introduced, which encodes a full-resolution image once and represents it with a mixture of native- and compressed-resolution visual tokens, and shows complementary bottlenecks in region selection and compressed-region fidelity.
Donghyun Han, Jangho Park, Yuseok Bae· 0 citations
An attention‐flip attack that manipulates pruning‐related attention redistribution by suppressing high‐importance visual tokens while promoting low‐importance regions, thereby increasing the likelihood that discriminative visual evidence is discarded during inference is proposed.
Shi-Kai Su, Hui Sun, Le-Feng Zhang· Concurrency and Computation· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.