Human–Object Interaction as Affective Evidence for Multimodal Emotion Recognition

Abstract

Emotion recognition is a key capability in human–robot interaction (HRI), but existing multimodal emotion recognition (MER) methods rely mainly on facial, vocal, and textual cues, which are often unreliable in practical HRI settings. In particular, occlusion and degraded or missing modalities can substantially reduce the performance of models that rely on conventional expressive modalities. This paper investigates human-object interaction (HOI), defined here as the observable interaction between a person and surrounding objects or scene elements, as a complementary modality for emotion recognition in HRI. To this end, we construct an HOI-aware emotion dataset of 464 short video clips, curated from HOIGen-1M, MER2024, and self-collected online videos, annotated with seven discrete emotion categories. We represent HOI using structured semantic descriptions of interaction attributes and integrate this modality into three representative MER baselines, namely Emotion-LLaMA, AffectGPT, and MMA-DFER. For Emotion-LLaMA and AffectGPT, HOI is introduced as additional semantic context in the textual input, whereas for MMA-DFER it is combined with the original model output through an emotion-wise gating-based fusion mechanism. Results show that HOI consistently improves all three baselines and provides particularly notable benefits under blurred-face conditions, while also yielding lower standard deviation across repeated runs in most settings. Overall, we demonstrate that HOI offers complementary cues that can enhance the robustness and stability of emotion recognition for embodied HRI.