Guided Temporal Modeling for Perceived Failure Detection in Human–Robot Interaction

Abstract

Robot failures are often expected when operating in dynamic human environments. Robot actions may be technically correct from the system’s perspective yet still violate users’ expectations, resulting in confusion, uncertainty, and interaction breakdowns. Detecting such perceived failures is therefore important for supporting effective human–robot collaboration, as it can enable adaptive explanations and interaction repair when expectation violations occur. In this paper, we address Track 1, Bystander Affect Detection (BAD), of the ERR@HRI 3.0 Challenge, which aims to detect whether participants are observing a failure or non-failure scenario from webcam recordings collected under naturalistic conditions. We propose a guided temporal framework that models facial behavior using a bidirectional LSTM with attention-based pooling. Here, ‘guided’ refers to auxiliary frame-level supervision derived from manually annotated temporal segments. The proposed temporal model for robot failure detection jointly performs frame-level and sequence-level prediction while incorporating consistency loss and temporal smoothness regularization to capture coherent behavioral dynamics over time. Evaluated on the BAD dataset, the proposed model achieved a macro F1-score of 0.567 and a balanced accuracy of 0.557 on the held-out test set, representing improvements of 6.5 and 5.3 percentage points over the official baseline in macro F1-score and balanced accuracy, respectively. These findings suggest the value of temporal facial modeling for detecting behavioral responses associated with user-perceived robot failures and may serve as one component of future adaptive explanation systems in human–robot interaction.