Real-Time Multimodal Emotion Recognition in Human-Robot Interaction: Deployment and Experimental Evaluation

Abstract

Emotion recognition is essential for natural and empathic human-robot interaction in socially assistive robots. However, real-time deployment remains challenging due to limited on-board computation, latency constraints, and the varying reliability of affective cues across modalities. In this work, we present a real-time multimodal emotion recognition approach for conversational interaction on a tabletop socially assistive robot. The approach integrates visual, audio, and textual cues and combines their outputs using a mixture-of-experts late fusion strategy to support inference when modalities are missing or unreliable during live dialogue. The robot's on-screen avatar dynamically adapts its facial expressions according to the system's estimated user emotion. Experiments include unimodal benchmarking on public video, audio, and text emotion datasets, as well as multimodal evaluation on the MELD dataset. A real-time study with seven participants further evaluates zero-shot and fine-tuned performance, as well as end-to-end latency on a resource-constrained robotic platform, demonstrating the feasibility of on-board multimodal emotion recognition while highlighting key deployment challenges in real-world human-robot interaction.