Enhancing Trust in Deepfake Detection: An Explainable Hybrid Framework Using Dual Behavioral and Contextual Cues

Abstract

Face-swapping and lip-sync manipulation tools have become easily available to the extent that identifying the difference between a fake video and a real one is no longer an easy task to a trained human analyst. We developed a three-branch architectural model, which includes EfficientNetB0 as a frame level spatial feature detector, a BiLSTM as the resultant time- varying features detector of blinking aberrancies, head motions aberrancies, and lip-sync errors, and a Vision transformer branch to examine the contextual consistency at the scene level. The 3 outputs are combined into a 3968-dimensional collection of numbers (26.77M total parameters) that are applied to a sigmoid classification. All the predictions are accompanied by grad-CAM overlays and a 0-100 percentage score allowing the system to have visual representation of which area of the face generated the decision and not just a label. On 60,000 images, the processing of images yielded a 88 percent accuracy which was better than a VGG16-only baseline accuracy of 78 percent and CNN+LSTM accuracy of 83 percent. Detection on 66 videos using 5-fold cross-validation was 100%. The system can be utilized in a real-life review process due to a deployed Flask REST API and a Streamlit front-end.