The Look of Error: Facial Indicators for Speech Recognition Failures

Abstract

Robots operating in human-centered environments frequently encounter interaction errors due to limitations of underlying machine learning models, potentially negatively affecting user trust and adoption. Therefore, detecting when the models fail is critical for robust HRI systems. Existing approaches mainly rely on model uncertainty (confidence) or supervised classifiers requiring extensive labeled data. In this work, we explore humans' affective reactions as implicit feedback signals for error detection by a robot. We develop a system to extract facial action units during situated verbal interaction with a robot and analyze data collected from 30 participants. Despite some individual variability in the reactions, a consistent pattern emerged across participants: lid tightening and left outer brow raising were more frequent during error situations. Although automatic speech recognition confidence scores provide a strong baseline for word-level error detection, combining them with facial cues significantly improves performance. Visual feedback of transcribed speech by the robot elicits more informative facial responses than spoken feedback. These findings highlight the benefit of combining implicit human feedback with model-based uncertainty for more accurate error detection, supporting the development of adaptive recovery mechanisms.