Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT
Experiments on ImageNet across multiple ViT architectures and attacks show that decoys can redirect high attention scores away from the true adversarial region while preserving much of the attack effectiveness, revealing a fundamental limitation of using attention magnitude as an indicator of adversarial relevance.