Detecting and Anticipating Robot Failures from Human Behavioral Signals

Abstract

Detecting robot failures from human behavioral responses is a challenging problem in human-robot interaction, due to the high inter-subject variability of overt reactions and the influence of contextual factors on their expression. In this work, we address two complementary formulations of this problem within the ERR@HRI 3.0 Challenge: detecting failure reactions from bystander videos; and anticipating perceived failures before they occur. For the former, we propose a decoupled two-stage framework in which a one-class deep metric learning model performs subject-calibrated pseudo-labeling of anomalous temporal windows, which are then used to supervise a spatio-temporal Graph Attention Network for fine-grained reaction classification. For the latter, we introduce a multimodal late-fusion Transformer that independently models visual appearance via a frozen EfficientNet-B0 backbone and facial landmark dynamics via frame-to-frame delta features, further fused with a Voice Activity Detection based audio score. Both approaches outperform the official challenge baseline across most primary evaluation metrics, demonstrating the effectiveness of our methods for detecting subtle behavioral signals in human-robot interaction. Code is available at https://github.com/eunyhwang/ERRHRI-3.0-PRISCA/.