MTDDAPT: Multi-Modal Text Distribution-Aware Deep Adversarial Prompt Tuning for Robust Vision-Language Representation Learning

Abstract

: Vision-language models (VLMs), such as CLIP, have achieved competitive performance in open-vocabulary visual recognition, but they remain vulnerable to adversarial examples. This vulnerability may affect the reliability of AI-based visual perception models when image inputs are perturbed. Existing robustness-oriented adaptation methods mainly optimize model weights or prompts, while the geometry of class-level textual representations remains underexplored during adversarial adaptation. In this work, we investigate clean accuracy and adversarial robustness of VLMs from the perspective of multi-modal text distribution optimization. We propose Multi-modal Text Distribution-Aware Deep Adversarial Prompt Tuning (MTDDAPT), which builds on multi-modal adversarial prompt coupling with frozen encoders and further introduces explicit distribution-aware regularization over class-level textual embeddings. Specifically, MTDDAPT uses a Gaussian-kernel-based inter-dispersion objective to enlarge inter-class separation in the normalized text embedding space, while PGD-based adversarial training optimizes deep textual prompts and text-to-visual projection mappings, thereby adapting the projected visual prompts. Across three visual recognition datasets, MTDDAPT consistently outperforms hand-crafted prompts, achieving average improvements of 19.51 and 11.44 percentage points in clean accuracy and robust accuracy, respectively. In the 16-shot setting, MTDDAPT further improves clean accuracy by 34.71 percentage points over hand-crafted prompts and achieves an average robust accuracy improvement of 22.32 percentage points across PGD, TPGD, and CW attacks. These results suggest that explicit text distribution-aware regularization can complement multi-modal prompt coupling and improve robust vision-language representation learning under adversarial image perturbations.