Evaluated on GTSRB and LISA across four backbones and three physical attack types, LAMDA is the only method among ten evaluated that consistently improves robustness across all attack-backbone-dataset combinations, while preserving or improving clean accuracy in nearly all cases.
Abstract
Traffic sign recognition (TSR) models based on deep neural networks achieve strong clean-data performance but remain vulnerable to physically realizable adversarial attacks, including shadow perturbations, natural-light interference, and printed patches. Existing defenses often improve robustness against one attack type while degrading performance on others, and can reduce clean accuracy. We propose LAMDA (Language-Anchored Model for Direction Alignment), a training framework that transfers language-grounded structure into TSR models without using adversarial examples or adding inference-time overhead. LAMDA builds two fixed prototype banks from VLM-generated sign descriptions and class names using a frozen OpenCLIP text encoder, and uses them to supervise visual features through two complementary auxiliary losses during training. At inference, the adapter and prototype banks are discarded, leaving a standard backbone and classifier. Evaluated on GTSRB and LISA across four backbones and three physical attack types, LAMDA is the only method among ten evaluated that consistently improves robustness across all attack-backbone-dataset combinations, with gains of up to +12.5 pp under shadow attacks and +13.2 pp under natural-light attacks, while preserving or improving clean accuracy in nearly all cases.
Traffic sign recognition systems are critical for autonomous vehicle safety yet remain vulnerable to adversarial perturbations that exploit environmental conditions. Existing naturalistic attacks employ simplified approximations without modeling underlying physics, while defenses lack comprehensive evaluation across diverse architectures and attack types. This work formulates six physically-grounded adversarial attacks spanning naturalistic perturbations (shadow, light patch, obstruction) and weather conditions (fog, snow, frost), optimized via Prior-guided Bayesian Optimization for black-box scenarios. We evaluate nine architectures across three geographically diverse datasets, revealing that transformer-based models exhibit 20.0 percentage points (pp) lower average attack success rate than CNNs (41.5% vs 61.5%) with comparable clean accuracy across datasets. We introduce Robustness Score (RS) to quantify resilience across all attacks, with baseline ConvNeXt-Tiny achieving 60.9% RS compared to best CNN at 50.7%. DiffPure diffusion-based purification substantially improves robustness through test-time noise injection and reverse denoising, increasing RS by 29.8-37.3 pp. Post-defense, transformer architectures achieve 6-16% residual attack success rate compared to 15-39% for CNNs, with consistent effectiveness across datasets (variance ⪅ 2.1 pp). These findings establish that architectural choice critically impacts adversarial robustness and that diffusion-based purification provides practical test-time defense for deployed autonomous driving systems.
Mubarak A. Alanazi· International Conference on...· 0 citations
While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks, posing significant security risks. Existing defense methods predominantly target single-task scenarios (e.g., zero-shot classification) and consequently lack generalizability across various multimodal tasks. To address this limitation, we propose a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals from two modalities, enhancing model robustness while generalizing across multiple downstream tasks. The proposed framework comprises two core components, i.e., $\textbf{Visual}$ supervision branch and $\textbf{Semantic}$ supervision branch. The former branch leverages features from clean images, extracted via a frozen original vision encoder, to guide adversarial robustness while the latter incorporates caption-image alignment as a contextual signal to preserve semantic coherence under attack. Moreover, our method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original model, with no need of separate task-specific retraining or architecture modifications.Extensive experiments demonstrate that our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering (VQA) tasks.
Sibo Wang, Jie Zhang, Shiguang Shan et al.· 0 citations
To efficiently generate adversarial examples, a gradient-based attack method is proposed that performs optimization exclusively on the vision encoder of the VLM rather than on the entire multimodal architecture, which significantly reduces the computational cost and resource requirements of the attack while maintaining strong effectiveness.
Ilan Zini, B. Addad, Katarzyna Kapusta· 0 citations
This work proposes RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment, and employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment.
Xingyu Zhu, Huanshen Wu, Shuo Wang et al.· 1 citation
AI-based visual perception systems are increasingly deployed in infrastructure surveillance, including roadside monitoring units, highway cameras, and smart-city pedestrian management systems. The security vulnerability of these systems to physical adversarial attacks poses a direct threat to the reliable operation of transportation infrastructure. We propose AdvSerial, a dynamic 2D--3D joint optimization framework for generating continuous high-angle physical adversarial patches against pedestrian detectors in infrastructure-based scenarios. We UV-map a boundary-aware quilted texture onto 3D garments, combine 2D digital attacks with 3D sparse- and continuous-frame rendering, and explicitly suppress person-specific semantic features while enforcing temporal continuity. A Feature Smooth Quilting strategy reduces visible patch boundaries and bounds cross-seam feature discontinuities. A serial-frame loss encourages long uninterrupted sequences of detection failures. In physical world experiments, AdvSerial achieves a 74.8% attack success rate on YOLO-v5 and degrades mean detection confidence from 84.30% to 39.38%. Experiments spanning eight detectors with different architectures demonstrate strong transferability. Notably, it achieves an $89.71%$ attack success rate on YOLO-v2 and resists both patch-detection defenses (NapGuard) and 3D-temporal perception (Sparse4D-v3). The results reveal persistent, temporally consistent failure modes under high-angle surveillance, and motivate the design of motion-aware and 3D-aware defenses for security-critical infrastructure deployments.
Yuanhao Huang, Yilong Ren, Jinlei Wang et al.· 1 citation
In recent years, with the rapid development of autonomous driving technology, lightweight neural networks (LNNs) have been increasingly applied in-vehicle and edge computing devices. A growing number of studies have focused on deploying LNNs in resource-constrained environments to achieve real-time traffic sign recognition, obstacle detection, and other tasks. However, while LNNs maintain high inference efficiency, their robustness has become an increasingly important concern. Research has shown that adversarial attacks targeting traffic signs can significantly disrupt the predictions of LNNs, leading to misclassifications. To enhance the adversarial robustness of LNNs, we propose TRADES-JR, a TRADES loss function guided by Jacobian regularization. This approach simultaneously considers the prediction distribution differences between clean and adversarial samples while penalizing the Frobenius norm of the network output’s Jacobian with respect to the input. By constraining the sensitivity to input perturbations, our method enhances the adversarial robustness of LNNs. We evaluate the proposed algorithm on the GTSRB and TSRD datasets. The results demonstrate that our method significantly improves the adversarial robustness of LNNs under both white-box and black-box attacks. Therefore, this method enables LNNs to maintain robust and high-accuracy traffic sign recognition even in adversarial environments, thereby enhancing the reliability of the autonomous driving system.
YunKai Zhao, Shang Gao, Jieliang Zhao· Proceedings of the Instituti...· 0 citations