Enhanced facial action unit detection with adaptable patch sizes on representative landmarks

Abstract

The human face displays expressions through the contraction of various facial muscles. The Facial Action Coding System (FACS) is a widely accepted taxonomy that describes all visible changes in the face in terms of action units (AUs). In this study, AUs are examined by finding the most active landmarks of the face and then examining the most representative patch sizes of each landmark for the AU detection task. Sparse learning is employed to learn the most active landmarks for each AU, and then the active landmark patches are fed to ViT and Perceiver mechanisms independently. Experiments indicate that using active landmark patches with their most representative size improves the results when compared to using all the landmarks, especially when it is used on more challenging datasets as a support for the attention mechanism of the classifier. The results demonstrate that the proposed method improves the performance of the employed models and are further supported by experiments conducted across different datasets. © 2025 Elsevier B.V., All rights reserved.