Exploiting Hierarchical Representations of Vision Foundation Models for Face Forgery Detection

Abstract

Generalizable face forgery detection has become a critical problem in multimedia forensics as modern face manipulation techniques can generate increasingly realistic facial content. Vision foundation models provide a promising basis for this problem, but existing detectors usually rely on the last-layer visual feature, implicitly assuming that the most transferable forgery evidence is concentrated in the final representation. In a vision transformer, different layers preserve different visual properties, and face manipulation traces may appear as local texture defects, region-level structural conflicts, or high-level semantic inconsistencies. We revisit CLIP from a hierarchical perspective and propose HIAF, a Hierarchical Interaction and Adaptive Fusion framework for face forgery detection. HIAF extracts representations from multiple transformer depths and learns to use them through two dedicated modules. Cross-Layer Feature Interaction (CFI) performs a bottleneck self-attention along the layer dimension, enabling each layer to absorb complementary evidence from other depths and exposing cross-level inconsistencies that are informative for forgery detection. Layer-aware Adaptive Fusion (LAF) then assigns dynamic weights to different layers, adaptively promoting the layers whose forensic cues are most reliable while suppressing less discriminative ones. To further stabilize the layer-wise feature spaces, we introduce a multi-view contrastive objective that treats each layer as an independent view and regularizes real samples into compact manifolds. Extensive experiments show that HIAF achieves an average AUC of 88.93% in cross-dataset evaluation and 96.54% in cross-manipulation evaluation, outperforming recent state-of-the-art methods.