Transformer-Based Multi-Modal Architecture for Real-Time Deepfake Detection in Streaming Video
Abstract
The rapid diffusion of generative deep learning models—specifically Diffusion-based video synthesis, Neural Radiance Fields (NeRFs), and advanced voice cloning autoencoders—has made synthetic media creation accessible to non-experts. While these techniques drive innovations in entertainment and virtual reality, they also enable severe threats, including identity fraud, automated disinformation campaigns, and financial manipulation via real-time streaming video feeds. Existing deepfake detection methods largely rely on single-modality unimodal classifiers (e.g., frame-by-frame Spatial Convolutional Neural Networks) that fail under dynamic streaming conditions due to frame drops, adaptive bitrate compression artifacts, and low latency budgets. This paper introduces an end-to-end Transformer-based Multi-Modal Architecture designed for real-time deepfake detection in live streaming video pipelines. By fusing spatial-temporal visual features (derived from a lightweight Video Vision Transformer, ViViT) with acoustic-phonetic speech representations (extracted via a self-supervised Wav2Vec 2.0 backbone) through a Cross-Modal Attention Bridge (CMAB), our model captures subtle audio-visual desynchronizations and lip-phoneme misalignments. To achieve real-time streaming throughput (>60 FPS at <45 ms end-to-end latency), we integrate TensorRT graph optimizations, structured token pruning, and a sliding-window temporal memory buffer. Benchmark evaluations across benchmark datasets (DFDC, FaceForensics++, and AV-Deepfake1M) demonstrate a superior Area Under the Curve (AUC) of 98.4% and an Equal Error Rate (EER) of 2.1%, outperforming unimodal and late-fusion baselines while operating efficiently on edge GPU infrastructure.