CFR-GAN++: Occlusion-Robust Multimodal 3D Face Recognition via Synergistic Visual-EEG-ECG Fusion
TL;DR
This study proposes a multimodal biometric framework, Complete Face Recovery (CFR)-GAN++, which combines self-supervised 3D face reconstruction with physiological biometrics from Electroencephalography (EEG) and Electrocardiography (ECG).
Abstract
Face recognition with occlusion remains a challenging issue for biometric authentication systems in real-world scenarios. Recent Generative Adversarial Network (GAN)- based approaches have improved facial reconstruction under partial occlusion; however, recognition accuracy remains severely limited in regions with extensive facial occlusion. To address this limitation, this study proposes a multimodal biometric framework, Complete Face Recovery (CFR)-GAN++. The framework combines self-supervised 3D face reconstruction with physiological biometrics from Electroencephalography (EEG) and Electrocardiography (ECG). The proposed framework consists of a facial reconstruction generator based on U-Net, a CNN–BiLSTM EEG encoder, and a 1D CNN ECG encoder in an adaptive feature-level fusion framework. The visual stream reconstructs occlusion-corrupted facial regions with a self-supervision strategy of Swap-Rotate-and-Render and 3D Morphable Model (3DMM) regression.