Deepfake tampering detection based on multimodal large model

Abstract

As deep learning generation technologies continue to evolve, Deepfake technology has become a major threat to information security, posing significant challenges to tampering detection. Existing detection methods for deep forgeries generally suffer from insufficient generalization capability and interpretability. To address these issues, this paper proposes a Deepfake tampering detection method based on a multimodal large model, termed LLM-FNP. This method integrates frequency-domain noise perception with visual-language understanding. First, the BayarConv module is employed to extract frequency-domain noise features from images, capturing anomalous noise patterns left by the forgery process, and a dual-branch encoder is used to process the original RGB images and frequency-domain feature maps separately. Second, a cross-attention mechanism is applied to achieve cross-modal fusion. Finally, the fused features are processed by a fine-tuned LLaVA large model, which outputs the detection result along with interpretable text. Experimental results on the DFDCP dataset show that the proposed method achieves an accuracy of 78.54% and an AUC of 81.50%. To evaluate the generalization capability of the method, cross-dataset testing on Celeb-DF and WildDeepfake yields accuracies of 73.5% and 71.8%, and AUCs of 76.4% and 75.1%, respectively, all outperforming classical methods such as ResNet+LSTM, further validating its strong generalization ability.