Language-Conditioned Audio Adaptation for Multilingual Audio-Visual Deepfake Detection

Abstract

Audio-visual deepfake detectors are increasingly deployed in multilingual settings, yet language-dependent speech variability can alter the behavior of audio branches trained without explicit language conditioning. This paper evaluates a MAVDD-style audio-visual detector on the French, Spanish, and Russian subsets of PolyGlotFake and proposes a lightweight language-conditioned MFCC adapter routed by predictions from a frozen MMS-LID-126 model. The Xception visual branch, the MFCC-CNN audio classifier, and the language identifier remain frozen; only 104 scalar adapter parameters are trained. To prevent source leakage, all manipulated variants derived from the same raw source are assigned to a single split. On the untouched 1,311-video test set, baseline hard-OR fusion obtains 91.15% accuracy, 62.58% balanced accuracy, 94.90% fake recall, and 95.28% fake F1. The language-aware system reaches 94.28% accuracy, 59.30% balanced accuracy, 98.87% fake recall, and 97.02% fake F1. The audio branch alone improves from 73.53% to 90.08% accuracy and from 59.86% to 66.31% ROC-AUC. Of 57 changed final decisions, 49 are baseline errors corrected and 8 are newly introduced errors; real recall falls from 30.26% to 19.74%. An exact paired McNemar test gives p ≈2.72 × 10−8. We also report that a trivial all-fake predictor attains 94.20% accuracy and 97.01% fake F1 on this test set, so the full-system accuracy gain should be read as increased fake sensitivity rather than improved class separation. Post-hoc Integrated Gradients, adapter-parameter visualization, and Grad-CAM provide qualitative evidence consistent with this trade-off. Index Terms—audio-visual deepfake detection, multilingual deepfake detection, language identification, MFCC, Xception, multimodal learning, language-conditioned adaptation, explainable AI.