Deepfake Detection via Frequency-Aware Vision Transformer and Bidirectional Cross-Attention Fusion with Post-Processing Robustness
TL;DR
FAViT (Frequency-Aware Vision Transformer), a hybrid architecture capable of jointly utilizing spatial- and frequency-domain forensic information by the means of a bidirectional cross-attention fusion scheme, is presented.
Abstract
Today, the use of increasingly ubiquitous synthetic media, or ‘deepfakes’, has become a risk to online trust, information integrity and individual security and is being created by artificial intelligence (AI). The current approaches are mainly based on either spatial features of CNNs or high-level semantic representations of Vision Transformer; both have major drawbacks in effectively leveraging multi-domain forensic cues. This paper presents FAViT (Frequency-Aware Vision Transformer), a hybrid architecture capable of jointly utilizing spatial- and frequency-domain forensic information by the means of a bidirectional cross-attention fusion scheme. We use an 11-channel forensic tensor in each face image (including per-channel Fast Fourier Transform (FFT) magnitude maps, Discrete Wavelet Transform (DWT) sub-bands, channel noise residual maps, Sobel gradient magnitude and channels of Error Level Analysis (ELA)). A Frequency Branch CNN processes this multi-domain tensor and the original RGB image is encoded with a pretrained ViT-B/16 spatial branch. The two streams are combined through the bidirectional cross-attention which allows the model to localize both spatial and spectral manipulation artifacts. We also present an adversarial cleaning simulation pipeline which partitions the training process with five post-processing attack methods, namely GFPGAN neural face restoration, learned autoencoder cleaning, etc., to increase resistance to real-world forensic defenses. Tests of FaceForensics++ C23 (7926 images, consisting of four manipulation types) show that FAViT attains F1-score of 86.22, AUC-ROC of 94.26 and accuracy of 85.55 on the held-out test set. The strength analysis of 21 attack conditions shows that the max degradation in AUC is 30.3, with specific strengths in GFPGAN restoration (AUC = 98.51). Robustness is evaluated based on 21 post-processing attack cases that include JPEG compression, Gaussian blurring, down-sampling, and GFDGAN neural-based restoration; it should be noted that robustness against gradient-based adaptive attacks requires additional attention. Testing on the CIFAKE and Celeb-DF v2 datasets reveals some limitations of domain generalization.