Skip to content
Book Open access

CaptchaBench: A Modality-Stratified Benchmark Dataset for Evaluating Adversarial Perturbation Against VLM-based CAPTCHA Recognition

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · pp. 16859-16865 · 0 citations · 8 references

TL;DR

CaptchaBench is introduced, a large-scale benchmark comprising 840,000 base Chinese-character CAPTCHA images (3,500 characters sampled from GB2312 Level-1 × 120 backgrounds × 2 generative pipelines), with a stratified evaluation subset of 16,000 adversarially perturbed images, confirming high human readability of perturbed CAPTCHAs.

Abstract

Vision-Language Models (VLMs) now achieve over 99% zero-shot accuracy on Chinese-character CAPTCHAs, rendering traditional hardening strategies (complex backgrounds, stylized fonts, character distortion) ineffective. Adversarial perturbation offers an emerging defense by exploiting model vulnerabilities at the decision-boundary level, yet existing methods originate from disparate communities and have never been compared under identical conditions; moreover, the role of perturbation input modality has not been systematically characterized across heterogeneous VLM architectures. To bridge these gaps, we introduce CaptchaBench, a large-scale benchmark comprising 840,000 base Chinese-character CAPTCHA images (3,500 characters sampled from GB2312 Level-1 × 120 backgrounds × 2 generative pipelines), with a stratified evaluation subset of 16,000 adversarially perturbed images. Eight perturbation methods are organized into three input-modality groups (Image-only: ASPL, Glaze, AMP, XTransfer, AnyAttack; Text-only: Nightshade; Image+Text: MMCoA, CoA) and assessed under a unified three-question framework: Q1 Direct-Recognition Accuracy (DRA) and Q1 Failure, Text Visibility Rate (TVR), and Q3 Attack Success Rate (ASR), capturing complementary black-box response behaviors. Baseline evaluation across five commercial VLMs (Qwen-VL, Kimi 2.5, GPT-5.2, Gemini-3.0, GLM-4V) yields three key observations: (1) perturbation input modality is associated with distinct black-box response patterns: image-only methods achieve high visual confusion but variable VLM-reported visibility, while methods differ in cross-VLM variation; (2) Gemini-3.0 exhibits a pronounced, model-specific TVR response pattern (62.4--97.3% vs. ?20.8% for all other VLMs; ≥3.0× vs. the highest non-Gemini TVR across all eight methods), that persists across prompt variants; and (3) stroke-bin Q3 ASR varies across methods, with the 16--20-stroke bin high in this evaluation; the sparsely sampled 21+ bin is reported descriptively. This motivates character structure as a CAPTCHA design dimension. A 29-participant four-choice study confirms high human readability of perturbed CAPTCHAs (98.9% accuracy). The dataset and evaluation code are released under CC BY 4.0. Extended tables, figures, and visual case studies are available as online supplementary material on the CaptchaBench dataset website at https://github.com/Linzy19/CaptchaBench.

Read PDF

Similar papers

Open access 2026

Annotation-Free Character Recognition for Azerbaijani License Plates: GAN Generator Diversity and Multi-Architecture Detection

This paper presents an automatic license plate recognition framework for Azerbaijani plates featuring annotation-free character recognition, addressing limited supervision, severe class imbalance, and unseen character combinations in low-resource settings. Unlike conventional approaches that require extensive character...

Javid Alakbarli, J. Hasanov, E. Mustafayev · 0 citations
Open access Oct 2026

IGAN: An Inception-Based Generative Adversarial Network for Multi-Scale Image Synthesis

Generative Adversarial Networks (GANs) provide a powerful framework for image synthesis, but their practical effectiveness is often limited by the interplay between high visual fidelity and stable adversarial training. Heavily parameterized GANs can improve feature representation; yet, they may also increase the risk o...

Ahmed A. Hashim, Ali Al-shuwaili, Asraa J. Saeed et al. · 0 citations
Preprint Oct 2026

A Unified Evaluation Framework of Physical Adversarial Robustness in the Edge VLM Era

Recent advancements in Vision-Language Models (VLMs) enable end-to-end visual grounding via language prompts and generative decoding, demonstrating largely improved zero-shot and open-vocabulary detection capabilities. As a result, VLMs are rapidly being deployed at the edge in safety-critical real-world systems. Howev...

E. Liang, Alan Luo, Kaiwen Yuan · 0 citations
Sep 2026

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

Vision-language pre-training has reshaped image clustering, giving rise to language-assisted image clustering (LaIC), which leverages textual semantics to complement visual representations. Despite the rapid proliferation of LaIC methods, it remains unclear how much LaIC has actually advanced image clustering, as exist...

Yuan Hu, Bo Peng, Yu-Heng Jia et al. · 0 citations
Open access 2026

MTDDAPT: Multi-Modal Text Distribution-Aware Deep Adversarial Prompt Tuning for Robust Vision-Language Representation Learning

The proposed Multi-modal Text Distribution-Aware Deep Adversarial Prompt Tuning (MTDDAPT), which builds on multi-modal adversarial prompt coupling with frozen encoders and further introduces explicit distribution-aware regularization over class-level textual embeddings, is proposed.

Chao Li, Zhen Tian, Cai-Chang Ding et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.