HARMONY: Multimodal Spectrum-Based Personality Adaptation for Human-Robot Interaction

Abstract

Personality plays a central role in human-robot interaction, shaping how people engage with social robots. Yet most existing systems treat personality as fixed or reduce it to binary categories, limiting adaptation across diverse users. Moreover, adaptation is often implemented at the level of isolated components, rather than through an integrated system that unifies perception, learning, and behaviour generation. Here, we present HARMONY (Hybrid Adaptive Robot personality through MultimOdal persoNality spectrum sYstems), a three-layer architecture that operationalises robot personality as a spectrum by integrating multimodal perception, hybrid offline-online reinforcement learning, and parameterised multimodal behaviour generation. HARMONY demonstrates how hybrid learning can mitigate the cold-start problem while supporting unobtrusive personalisation, and how parameterised behaviours enable consistent adaptation across verbal and non-verbal modalities. We validate the system in a user study showing that HARMONY sustains significantly longer interactions than a random baseline and eliminates disparities in engagement across users’ personality types. These findings provide evidence that integrated, hybrid, multimodal architectures can support socially responsive personality adaptation in HRI.