PAST: Prior-Aware Sparse Transformer for Micro-Expression Recognition

Abstract

Micro-expression recognition (MER) has a lot of applications in lie detection, education, healthcare, etc., as involuntary micro-expressions (MEs) may provide subtle facial cues associated with affective responses. With the development of deep learning, many studies have recently employed Vision Transformers (ViTs) to investigate MER, since ViTs show promising performance in various visual domains due to their excellent local–global modeling ability. However, such methods confront two fundamental challenges: First, fine-grained visual features are needed to capture the subtle facial movements of MEs, which ViTs relatively fall short on due to coarse patch resolution constrained by their quadratic complexity. Second, the data-intensive nature of ViTs impedes effective learning given the limited scale of ME data. To overcome the aforementioned limitations of using ViTs for MER, we propose the Prior-aware Sparse Transformer (PAST), a novel Transformer-based architecture integrating spatial and semantic prior knowledge synergistically into a sparse attention mechanism, enabling linear-complexity processing of large amounts of fine-grained features. Specifically, we first designed an extraction algorithm to generate a representative set of motion-intensive Principal Anchors, which are used to guide the model’s focus on biologically critical regions during sampling. Second, we introduced the Semantic Dictionary, which was trained with a carefully designed self-contrastive loss to embed task-invariant discriminative semantics of the anchors. Such global semantics further modulate patch sampling and attention weighting in the sparse attention procedure, achieving better training performance with limited ME data. Extensive evaluations on MEGC and CD6ME protocols demonstrate state-of-the-art performance, validating PAST’s efficacy for MER.