Skip to content

Backdoor as Probe: Test-Time Adversarial Defense for CLIP

Sep 2026 · 0 citations
Computer Science

TL;DR

Backdoor as Probe is proposed, a test-time adversarial defense for CLIP that improves average robust accuracy from 1.0\% to 52.3\% while retaining clean accuracy, achieving performance comparable to state-of-the-art methods with up to a \(5.7\times\) inference speedup.

Abstract

Test-time adversarial defense improves the robustness of vision-language foundation models such as CLIP without retraining. However, adversarial activation shifts are typically treated as distortions to suppress, rather than signals to exploit. We turn these shifts into defense signals by repurposing the trigger-to-target mechanism of backdoors. The key is to implant a defender-controlled backdoor as a probe that is weakly activated by clean inputs but strongly activated by adversarial shifts. Based on this insight, we propose \emph{Backdoor as Probe} (BaP), a test-time adversarial defense for CLIP. BaP constructs the probe through a closed-form model edit to a selected MLP layer. It projects the average adversarial activation shift and a defender-specified semantic direction onto the layer's low-energy input and output activation subspaces to obtain the trigger and target directions, respectively. At inference time, adversarial inputs produce measurable responses along the target direction for detection. BaP then selectively rectifies detected inputs by optimizing a small perturbation that steers their representations away from adversarial shifts and toward the clean subspace. Experiments across 16 benchmarks show that BaP improves average robust accuracy from 1.0\% to 52.3\% while retaining clean accuracy, achieving performance comparable to state-of-the-art methods with up to a \(5.7\times\) inference speedup. BaP further shows the generalization to adversarial attacks on large vision-language models. Project page: https://robin-wzq.github.io/Backdoor-as-Probe/

View source

Similar papers

#artificial intelligence Preprint Sep 2026

What Makes Adversarial Examples Transfer Across Deepfake Detectors?

A controlled evaluation of adversarial transferability across 60 detectors spanning six backbones, two pretraining regimes, and five training-data configurations establishes source-model selection as central dimensions of credible transfer-based black-box robustness evaluation.

Rafael M. Mamede, Pedro C. Neto, A. F. Sequeira · 0 citations
Conference Open access Sep 2026

Understanding and Exploiting Phase Sensitivity for Attacking Large Vision–Language Models

This paper proposes a novel LVLM attack method, called BadPhase with further backdoor designs, to implant adversarial phase as triggers into any image inputs via data poisoning so as to control the LVLMs’ predictions and finds that LVLMs are sensitive to the phase-aware image structure.

Dai-Zong Liu, Junhao Dong, Xiang Fang et al. · 0 citations
Open access Sep 2026

PatchGuard-Freq: Zero-Overhead Adversarial Patch Defense via Frequency Detection and Data-Driven Robustness

This work characterizes the complementary relationship between reconstruction-based and robustness-based paradigms in the accuracy–efficiency design space under the evaluated conditions: the former suits compute-unconstrained scenarios while the latter serves latency-constrained deployments.

De-Jie Luan, Cheng-Hua Li, Chun-Jie Zhang et al. · 0 citations

One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs

This work proposes O-Attack, a highly transferable black-box attack framework that outperforms six state-of-the-art methods in black-box transferability, with consistent effectiveness across prompts and improved efficiency and imperceptibility.

Sen Nie, Jie Zhang, Zhong Ling Wang et al. · 0 citations
Sep 2026

Combating Overfitting of Adversarial Training Efficiently via Balanced Instance Adaptive Defense

Adversarial training, one of the most effective methods for enhancing neural network robustness, is typically formulated as a min-max game between an attacker and a defender. Despite its success, most adversarial training methods suffer from robust overfitting, leading to a significant gap in robustness between the tra...

Xin-Yue Zhang, Shao-Cong Wu, Qiben Shan et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.