TL;DR
Sketch-text-driven rectified flow (STDRF), a conditional rectified-flow framework for identity-preserving and semantically controllable 4D face generation, and a sketch encoder enhanced by Geometric Contour and Texture Detail preprocessing and MixStyle domain adaptation are proposed.
A diffusion-based framework with a stage-wise multi-condition guidance mechanism that enhances both structural and textural fidelity and compares with recent image-to-image translation and diffusion-based baselines to observe competitive performance in both visual coherence and identity preservation.
Text-driven face understanding fundamentally depends on the quality of semantic descriptions; however, most existing face datasets provide only coarse or generic captions, limiting both retrieval accuracy and controllable face synthesis. Despite recent advances in multimodal generative models, face-centric applications continue to suffer from two critical challenges: reliable image retrieval from textual descriptions and identity-preserving semantic editing. In this work, we propose a modular yet unified framework that jointly addresses these challenges through enhanced semantic grounding and controlled generative modeling. First, we leverage LLaVA to automatically transform basic annotations in the dataset into rich, fine-grained facial descriptions through a multimodal semantic distillation process, significantly strengthening text–image alignment and improving large-scale text-based face retrieval. Building upon this semantic foundation, we introduce CITA-Net, a novel identity-aware face editing architecture based on Stable Diffusion XL (SDXL). CITA-Net employs an attribute-inversion data construction strategy derived from CelebAMask-HQ, enabling precise semantic disentanglement and controllable facial edits while preserving subject identity. Extensive experiments demonstrate that CITA-Net achieves a superior balance between editability, identity preservation, and visual fidelity, outperforming competitive diffusion-based baselines. In particular, our model attains lower identity loss, enhanced semantic alignment, and improved image quality. By unifying enhanced retrieval with photorealistic, identity-preserving face editing within a single framework, this work establishes a strong foundation for prompt-driven face understanding and enables practical applications such as forensic retrieval, semantic face editing, and natural language-based image search.
Text-driven 3D face generation and editing remains challenging due to the difficulty of translating long-form descriptions into fine-grained facial geometry. Existing methods primarily align global textual semantics with facial structures but often struggle to capture subtle local deformations, such as eyebrow tension, cheek contraction, and asymmetric mouth motions, resulting in limited geometric fidelity and editing precision. To facilitate fine-grained text-driven facial modeling, we first construct FaME-G2E, a large-scale multimodal dataset containing detailed text--mesh annotations and paired text--blendshape samples for unified 3D facial generation and editing. Based on this dataset, we propose RAGMesh, a retrieval-augmented framework that leverages text-correlated geometric priors to improve high-fidelity facial synthesis and editing. Specifically, the Multi-Scale Retrieval Fusion (MSRF) module retrieves semantically consistent global and regional facial priors and fuses them in the blendshape space, suppressing conflicting local deformations while preserving coherent deformation patterns. Furthermore, we introduce Adaptive RAG-guided Supervision (AdaRAGS), a region-aware constraint that explicitly aligns textual semantics with corresponding facial regions, enhancing regional controllability and editing accuracy. Extensive experiments on FaME-G2E demonstrate that RAGMesh achieves superior performance over state-of-the-art methods in local geometric accuracy, text-guided controllability, regional editing precision, and inference efficiency. Video demo is available at https://youtu.be/Yr0_XkpWcNk, and the source code and dataset will be released upon paper acceptance.
Forensic sketching translates a witness description into a visual representation of a suspect, but sketches often lack visual details, which can limit their use in face recognition systems. This paper presents a sketch-guided face generation pipeline based on a Conditional Variational Autoencoder (CVAE) designed to generate facial reconstructions from sketches and conditional attributes. The method uses a stochastic preprocessing pipeline to extract edge maps from facial photographs, reducing dependence on manually paired sketch-photo datasets. Conditional inputs are incorporated to control ambiguous attributes that may not be fully specified by the sketch. The proposed approach is evaluated using synthetic edge maps, hand-drawn sketches, and digitally drawn sketches from different sources, considering both reconstruction quality and identity-oriented similarity. Compared with a vanilla autoencoder, the proposed CVAE reduced the FaceNet distance by 56.8% in the CUHK reconstruction evaluation. The experimental results also suggest that considering both reconstruction quality and identity preservation may provide a more complete evaluation of sketch-guided face generation in forensic scenarios.
Neural Radiance Fields (NeRF) have enabled photorealistic novel-view synthesis of 3D scenes and, in the facial domain, have been extended to reconstruct and animate 3D face models from a small number of images. However, existing few-shot dynamic NeRF methods for facial expression editing typically warp a single learned feature volume conditioned on target expression parameters, which can cause identity-specific appearance details (skin texture, fine geometric structure) to drift when the model is driven toward expressions far from those seen in the few-shot input set. We propose Identity-Consistent Expression Fields (ICEF), a framework that explicitly disentangles a static, identity-specific radiance component from a dynamic, expression-conditioned deformation component, and introduces an identity preservation regularizer that constrains the deformation network to modify only expression-relevant regions while leaving identity-specific canonical appearance untouched. ICEF further incorporates a confidence-weighted conditional feature warping step that down-weights unreliable warps for target expressions that are far, in parameter space, from the observed few-shot inputs, mitigating artifacts observed in prior few-shot dynamic NeRF methods when extrapolating to novel expressions. We relate ICEF to prior few-shot dynamic NeRF, static 3D-aware face generation, and disentangled face-editing radiance field methods, and describe an evaluation protocol measuring both novel-expression rendering quality and, specifically, identity-consistency metrics across a range of expression-parameter extrapolation distances.
Generative diffusion models have revolutionized facial image synthesis, yet robust identity preservation in high resolution outputs remains a critical challenge. This issue is especially vital for security systems, biometric authentication, and privacy sensitive applications, where any drift in identity integrity can undermine trust and functionality. We introduce Diff-ID, a diffusion based framework that enforces identity consistency while delivering photorealistic quality. Central to our approach is a custom 210K image dataset synthesized from CelebA-HQ, FFHQ, and LAION-Face and captioned via a fine tuned BLIP model to bolster identity awareness during training. Diff-ID integrates ArcFace and CLIP embeddings through a dual cross attention adapter within a fine tuned Stable Diffusion UNet. To further reinforce identity fidelity, we propose a pseudo discriminator loss based on ArcFace cosine similarity with exponential timestep weighting. Experiments on held out and unseen faces show that Diff-ID does not exceed InstantID in raw ArcFace Face Similarity, but achieves substantially lower FID and the strongest FIQ based identity--realism trade off among the evaluated methods. We also present a unified DDIM based morphing pipeline that enables qualitative facial interpolation without per identity fine tuning. We further argue that identity preservation and photorealism should be evaluated jointly rather than in isolation, as high identity similarity alone does not guarantee realistic outputs. To make this trade off explicit, we report Face Image Quality (FIQ) as a complementary ratio based score that combines identity similarity and perceptual realism while keeping FS and FID as the primary metrics.