Affect-aware conversational adaptation in mixed reality procedural tasks

Abstract

Conversational agents based on Large Language Models (LLMs) are increasingly explored in eXtended Reality (XR) applications that require real-time guidance during task execution. In task-oriented XR environments, conversational support must remain responsive and integrated with the ongoing activity, minimizing interruptions in user interaction. Although affect-aware adaptation has been proposed as a possible strategy to improve interaction quality, its impact in immersive task-oriented scenarios remains unclear. This paper presents a mixed-reality conversational system that integrates voice interaction, LLM-driven dialogue, and prosody-based affect-aware adaptation. The system adopts a modular client–server architecture with parallel semantic and affective processing pipelines, enabling emotion-related cues inferred from vocal prosody to be incorporated without interrupting conversational flow. The system was evaluated through a between-subjects study involving 40 participants performing a guided chemical procedure within a mixed-reality laboratory scenario. Participants interacted with either an affect-adaptive or a non-adaptive version of the conversational agent. The evaluation combined standardized questionnaires, behavioral metrics extracted from interaction logs, and qualitative feedback. Results showed that the affect-adaptive condition produced greater perceived usability scores compared to the non-adaptive condition. Participants interacting with the adaptive agent also required longer task completion times and engaged in a higher number of conversational turns. No significant differences emerged for overall workload between the two conditions. Qualitative findings further suggested that affect-aware adaptation made the interaction feel more natural and supportive.