RVUF: an identity-aware and scene-cut-aware framework for robust character-centric video understanding

Abstract

The rapid growth of interactive video applications, including high-quality visual effects and character-centric interactions, requires video understanding systems to jointly perform face detection, face recognition, facial landmark and head-pose estimation, and instance segmentation. In film and television content, applying all task models to every frame and detected person wastes computation on redundant frames and background characters, while motion blur and occlusion lead to missed detections that fragment target-character trajectories and cause character-anchored effects to flicker. We propose the Robust Video Understanding Framework (RVUF), which takes a video and reference face images as input and produces identity-consistent target trajectories containing face bounding boxes, facial landmarks, head poses, and instance masks. RVUF first employs an identity-aware multi-task pipeline that selects informative keyframes, propagates results to redundant frames, and filters non-target candidates before computationally intensive inference. It then applies scene-cut-aware gap filling, using object tracking and recovered-box verification to locate missing targets and reconstruct their task-specific metadata. Experiments on real-world film and television data show that RVUF improves trajectory coverage, identity consistency, and spatial metadata quality while maintaining real-time throughput. RVUF has been deployed on the Youku video platform, providing a cost-effective solution for character-centric interactive video services.