Spatial Transformer Networks for Facial Action Unit Detection: Modeling Inter-AU Dependencies via Structured Attention

Abstract

Facial Action Unit (AU) detection is fundamental to understanding human facial expressions and emotions. While Transformers have revolutionized computer vision through self-attention mechanisms, their application to AU detection has primarily followed sequential processing paradigms. In this paper, we present a novel Spatial Transformer framework that reimagines AU relationship modeling through structured spatial attention mechanisms. Unlike conventional Vision Transformers that process images as sequences of patches, our approach treats AUs as spatially and functionally related entities, leveraging anatomically-informed attention patterns to capture complex inter-AU dependencies. The key innovation lies in our structured attention mechanism that incorporates multi-dimensional relationship encodings, enabling the model to learn both co-occurrence patterns and mutual exclusion constraints among AUs. We introduce spatial position encodings based on facial muscle locations rather than learnable embeddings, providing stronger inductive biases for facial analysis tasks. Our comprehensive experiments demonstrate that the Spatial Transformer effectively captures subtle AU relationships that are often missed by independent detection approaches. The learned attention patterns align remarkably well with established facial anatomy knowledge and FACS-defined AU relationships, offering interpretable insights into the model’s decision-making process. Furthermore, our analysis reveals that spatial attention mechanisms are particularly effective for detecting AUs with low activation intensities and those that frequently co-occur with other units. This work not only advances the state-of-the-art in AU detection but also provides a new perspective on modeling structured relationships in facial analysis through the lens of spatial attention, opening avenues for more sophisticated facial behavior understanding systems.