Secure Hardware Accelerator for Resilience Against Adversarial Attacks in ViTs

Abstract

Vision Transformers (ViTs) have demonstrated exceptional performance across various computer vision tasks. However, their susceptibility to adversarial white-box attacks poses a significant risk, especially in safety-critical applications. These attacks, with full model access, pose severe threat by drastically reducing accuracy. In response, we propose SAMURAI, a lightweight, preprocessing-based defense pipeline that mitigates adversarial perturbations without requiring model retraining or architectural changes. SAMURAI defends against adversarial inputs by applying a fixed sequence of four transformations: JPEG compression, resizing and center cropping, Gaussian smoothing, and normalization. This process disrupts adversarial attacks while preserving the semantic integrity of the image. Unlike computationally expensive defenses, SAMURAI introduces minimal overhead (e.g., 0.001x for ViT-Large), making it extremely lightweight for real-time deployment. Furthermore, our method generalizes effectively to other vision architectures, such as Convolutional Neural Networks (CNNs), demonstrating strong transferability. To support efficient deployment in resource-constrained environments, we also design a custom hardware accelerator for the SAMURAI pipeline that enables in-situ preprocessing within accelerators, effectively mitigating adversarial attacks. Experimental results show that SAMURAI achieves robust accuracy comparable to state-of-the-art while requiring only 0.009x the computational cost.