Skip to content
Open access

Transformer-Based Multi-Modal Architecture for Real-Time Deepfake Detection in Streaming Video

Oct 2026 · International Journal of Research Publication and Reviews · 0 citations

Abstract

The rapid diffusion of generative deep learning models—specifically Diffusion-based video synthesis, Neural Radiance Fields (NeRFs), and advanced voice cloning autoencoders—has made synthetic media creation accessible to non-experts. While these techniques drive innovations in entertainment and virtual reality, they also enable severe threats, including identity fraud, automated disinformation campaigns, and financial manipulation via real-time streaming video feeds. Existing deepfake detection methods largely rely on single-modality unimodal classifiers (e.g., frame-by-frame Spatial Convolutional Neural Networks) that fail under dynamic streaming conditions due to frame drops, adaptive bitrate compression artifacts, and low latency budgets. This paper introduces an end-to-end Transformer-based Multi-Modal Architecture designed for real-time deepfake detection in live streaming video pipelines. By fusing spatial-temporal visual features (derived from a lightweight Video Vision Transformer, ViViT) with acoustic-phonetic speech representations (extracted via a self-supervised Wav2Vec 2.0 backbone) through a Cross-Modal Attention Bridge (CMAB), our model captures subtle audio-visual desynchronizations and lip-phoneme misalignments. To achieve real-time streaming throughput (>60 FPS at <45 ms end-to-end latency), we integrate TensorRT graph optimizations, structured token pruning, and a sliding-window temporal memory buffer. Benchmark evaluations across benchmark datasets (DFDC, FaceForensics++, and AV-Deepfake1M) demonstrate a superior Area Under the Curve (AUC) of 98.4% and an Equal Error Rate (EER) of 2.1%, outperforming unimodal and late-fusion baselines while operating efficiently on edge GPU infrastructure.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.