Evaluating User Perception and Experience of Persona-Driven LLM-Powered Social Robots

Abstract

Large language model (LLM)-integrated social robots have the potential to be adopted in new application domains thanks to recent advances in conversational and behavioural capabilities. Their ability to engage in natural dialogue and adopt distinct personas makes them suitable for applications such as conversational practice, where robots can simulate human interaction scenarios, including socially complex communication contexts. However, deployment in real-world settings requires a clear understanding of user perceptions and preferences, as well as LLMs' capabilities and limitations with persona-driven prompts. In this paper, we evaluated 48 participants' attitudes towards a social robot embodying four distinct personas (two for career advising and two for diet advising), assessed user experience, and examined whether participants' perceptions of these personas influence their experience. The study used a mixed-subject design, with one group role-playing as career advisors and the other as diet advisors. The findings provide insights into the potential of LLMs to support distinct behavioural traits through careful persona and prompt design, and how these shape user experience and perceptions in the development of student training applications. The results suggested that the users' perceptions were consistent with the intended persona design. The system's usability was rated highly, and statistical analyses indicated that participants' individual traits influenced their perceptions of the system. These results will inform our future work in the novel application area of using AI-powered social robots as communication practice tools for Optometry university students.