A novel fine-grained identity-aware transformer (FIT) method for generalizable diffusion face forgery detection (DFFD) motivated by the novel observation that the preservation of target identity in facial images generated by GAN and diffusion models varies significantly.
Abstract
Face forgery detection (FFD) is essential for the security and authenticity verification of digital media. Current FFD methods suffer from poor generalization to unseen face images created by diffusion models. Besides, they tend to rely on coarse-grained prior information interaction paradigms. In this paper, we propose a novel fine-grained identity-aware transformer (FIT) method for generalizable diffusion face forgery detection (DFFD). Specifically, we are motivated by the novel observation that the preservation of target identity in facial images generated by GAN and diffusion models varies significantly. We employ the inherent identity preservation differences between GAN and diffusion face images to capture identity-aware forgery representations in a fine-grained learning manner. We employ the learned identity forgery embeddings as prior information to facilitate DFFD. We propose a fine-grained identity-aware transformer block (FITB) to mine fine-grained global identity-appearance forgery features based on intra-patch identity-aware relations as well as inter-patch global identity-perceptual relationships in diffusion face images. An identity contrastive center loss is devised to achieve intra-class identity forgery embedding compaction and inter-class identity forgery representation separation, to study discriminative and general diffusion face forgery patterns. Extensive experimental results demonstrate that FIT outperforms the state-of-the-art via cross-generator, cross-dataset, and robustness evaluation.
Image Manipulation Localization (IML) aims to precisely identify forged regions within an image at the pixel level, which is crucial for preventing the spread of misinformation that may pose potential threats to public safety. Recent Transformer-based methods can model long-range dependencies to capture global forgery features, however, they usually require heavy computation and overlook local fine-grained details that are essential for forensic analysis. To address these limitations, we propose a Fine-grained Forgery-aware Mamba (F2Mamba) that can efficiently and accurately localize potential manipulations. Specifically, F2Mamba learns multi-scale global features at low computational cost with the linear-complexity global modeling capability of VMamba. The Fine-grained Forgery-aware Adapter (FFA) is further introduced to adaptively fuse local critical details with global representations during decoding, which facilitates the recovery of fine-grained information lost in the earlier feature learning stage. Additionally, a Forgery-Guided Refinement Decoder (FRD) is designed at the final decoding stage to calibrate boundary errors by performing iterative Conditional Random Field (CRF) refinement, effectively suppressing checkerboard artifacts and improving localization precision. Extensive experiments on diverse datasets show that F2Mamba achieves an average F1 score of 69.92% and an average IoU score of 61.26%, demonstrating strong generalization capability compared with state-of-the-art IML models. Code is available at: https://github.com/ii-zy/F2Mamba.
Zi-Ying Zhao, Nan-Run Zhou, Yan Luo et al.· Neural Networks· 0 citations
Generative diffusion models have revolutionized facial image synthesis, yet robust identity preservation in high resolution outputs remains a critical challenge. This issue is especially vital for security systems, biometric authentication, and privacy sensitive applications, where any drift in identity integrity can undermine trust and functionality. We introduce Diff-ID, a diffusion based framework that enforces identity consistency while delivering photorealistic quality. Central to our approach is a custom 210K image dataset synthesized from CelebA-HQ, FFHQ, and LAION-Face and captioned via a fine tuned BLIP model to bolster identity awareness during training. Diff-ID integrates ArcFace and CLIP embeddings through a dual cross attention adapter within a fine tuned Stable Diffusion UNet. To further reinforce identity fidelity, we propose a pseudo discriminator loss based on ArcFace cosine similarity with exponential timestep weighting. Experiments on held out and unseen faces show that Diff-ID does not exceed InstantID in raw ArcFace Face Similarity, but achieves substantially lower FID and the strongest FIQ based identity--realism trade off among the evaluated methods. We also present a unified DDIM based morphing pipeline that enables qualitative facial interpolation without per identity fine tuning. We further argue that identity preservation and photorealism should be evaluated jointly rather than in isolation, as high identity similarity alone does not guarantee realistic outputs. To make this trade off explicit, we report Face Image Quality (FIQ) as a complementary ratio based score that combines identity similarity and perceptual realism while keeping FS and FID as the primary metrics.
T. Rizwan, Sara Atito, Muhammad Awais et al.· 0 citations
Deepfake technology has facilitated the spread of malicious content, creating an urgent need for efficient and accurate face forgery detection techniques. Traditional detection methods usually follow a two-stage paradigm, where faces are first detected and then classified as real or fake. However, such approaches can not simultaneously handle multiple faces, and errors in the first stage directly affect the second stage, leading to error accumulation. In this paper, we propose an end-to-end Transformer-based framework, termed Progressively Explicit Query Network (PEQNet), for multi-face forgery detection and localization. Our method extracts instance-aware activation queries from multi-scale, cross-domain features, and jointly processes them with image features through a progressively explicit query-driven decoding network to predict query-dependent masks and classes. During training, we introduce triple contrastive learning to model the mutual exclusivity among real, fake, and background regions. Additionally, we define specific real-or-fake queries to capture global class-level information, which are iteratively refined by the predicted class-aware masks and optimized by aligning them with the ground-truth. Extensive experiments demonstrate that PEQNet achieves superior performance compared with state-of-the-art methods, confirming its effectiveness for multi-face forgery detection in realistic scenarios.
Pengwen Dai, Xiaomeng Wen, Feiyang He et al.· ACM Transactions on Multimed...· 0 citations
The rapid evolution of face forgery techniques has introduced an increasing variety of manipulations. Incremental Face Forgery Detection (IFFD), which incrementally adds new forgery data to fine-tune previously trained models, has emerged as a promising approach to handle evolving forgery threats. However, conventional replay-based IFFD methods suffer from catastrophic forgetting. Storing full historical images under limited memory often either fails to preserve subtle forgery cues or introduces domain bias, reducing the model's ability to learn intrinsic and transferable manipulation characteristics. In this paper, we propose a Density-Aware Regional Decisive replay strategy, termed InfoDense, to address these challenges. InfoDense prioritizes artifact-dense and forgery-critical regions, significantly reducing storage requirements while maintaining high-fidelity forgery evidence. We first introduce InfoDense Cut to localize decisive patches using CLIP-based embeddings. Then, InfoDense Select ranks candidate segments by combining latent-space representativeness and decisive patch counts, ensuring both diversity and information density in the replay buffer. Finally, InfoDense Fuse reconstructs unbiased training inputs by adaptively merging stored segments with current-task samples, enhancing knowledge retention and generalization. Extensive experiments on challenging incremental deepfake benchmarks demonstrate that InfoDense effectively mitigates catastrophic forgetting while improving cross-domain generalization.
Jikang Cheng, Hao Shen, Xueyi Zhang et al.· 0 citations
This work proposes a robust, generalizable proactive face-swapping defense via semantic gradient divergence (SGD-Guard), and introduces an integrated feature gallery that uses CLIP features and a generalized identity feature, obtained by iteratively refining heterogeneous identity features into a homogeneous representation.
Do Seung-hyeok Back, Hyun Ki, Juwan Kim et al.· 0 citations
Generative AI and synthetic media generation tools have enabled widespread media manipulation tools and raised important privacy concerns with misinformation, identity fraud and the verification of authenticity of media. Most of the current convolution-based deepfake detection methods are hard to be deployed in real scenarios and hard to be interpretable, especially because they have limited ability to capture long-range spatial dependency. It introduces an explainable deepfake face image detection framework based on a vision transformer network for performing powerful binary classification of manipulated and real facial images and an explainable face image localization framework for localizing deepfake image faces. The proposed system involves a transformer-encoder backbone for extracting features through a patches-wise process, which proves suitable for modeling the subtle changes of features when the processes of manipulating the image are designed. To make the network more interpretable, and aid the understanding of the transformer attention distribution as well as localization of manipulated facial regions, a dedicated attention rollout mechanism is embedded. A dedicated rollout mechanism for attention distribution of the transformer and heatmap generating and attention spatial localization are incorporated to improve the interpretability of the network. The framework comprises an end-to-end inference pipeline, such as image preprocessing, estimation of confidence scores, fake-real classification, generation of explainable visualization and storage of prediction history using an integrated database system. An experimental evaluation shows the system can effectively detect deepfakes while also providing accurate local information as justification for classification decisions, contributing to transparency, reliability and trust towards automated synthetic media detection systems.
K. Phani, Shaik Mahaboob, Jailan PG Student et al.· 2026 International Conferenc...· 0 citations