Arab-DF: An Arabic-Media Visual Deepfake Dataset and a Reproducibility-Controlled Cross-Dataset Benchmark

Abstract

Deepfake detectors are rarely evaluated on Arabic-media content. We introduce Arab-DF, an Arabic-media visual deepfake dataset of 1066 authentic and 11,240 manipulated videos covering four manipulation families. Thirteen public detector checkpoints are evaluated zero-shot under a common protocol, and each must first reproduce its DeepfakeBench Celeb-DF-v2 frame-level AUC within ±0.02; eight meet this criterion. On the fixed 824-video subset, their mean video-level AUC falls from 0.820 on Celeb-DF-v2 to 0.552 on Arab-DF (ΔAUC = −0.267), and five have bootstrap 95% confidence intervals above 0.5. When the authentic sample is enlarged from 75 to 392 while retaining 749 manipulated videos, the mean is 0.508 (95% CI [0.478, 0.537]); only EfficientNet-B4 retains a 95% confidence interval entirely above 0.5. Lip synchronization has the lowest family-level point estimate, although intervals overlap. Seven file-level variables separate the classes (out-of-fold AUC 0.895), yet normalization on a 94-video subset leaves mean detector AUC essentially unchanged (ΔAUC = −0.0008; 95% CI [−0.054, 0.042]). Effort reaches 0.695 on the fixed subset but fails the control and is reported separately. Overall, reproducible public checkpoints transfer poorly to this fixed Arabic-media evaluation domain.