When Explanations Shape Trust: Challenges for Embodied AI Systems

Abstract

Technology advancements are leading to a future in which service robots are fully integrated into everyday environments. As these robots transition to being powered by LLMs, their inherently non-deterministic behavior can generate responses that appear correct and well-reasoned, even when they are not, raising challenges for the development of trust and reliance. While explanations have been widely shown to improve user acceptance and appropriate interaction in human-computer interaction, their role in embodied systems, particularly under conditions of uncertainty and error, remains insufficiently explored. In this work, we investigate the effects of natural language justifications on trust, compliance, and behavior in a robot-assisted recycling task using a controlled within-subject design. We find evidence that providing explanations for decisions significantly increases user compliance and reliance, but simultaneously introduces non-trivial trade-offs. Our results indicate that explanations can impair users' ability to critically assess and override incorrect recommendations. Moreover, their impact on trust over time is not consistent; it varies depending on users' prior knowledge and the difficulty of the task. Finally, we observed that explanations influence the speed of decision-making, with participants acting faster when explanations were provided. These findings highlight that explanations in embodied AI systems act not only as tools for transparency but as mechanisms that introduce fundamental challenges for designing trustworthy and well-calibrated human-robot interactions.