Visualizing Multimodal UX and Human Motion data in 3D scenes

Abstract

Understanding user experience (UX) in immersive and interactive media increasingly involves rich embodied interactions, where human motion, encompassing both human position and human body kinematics, evolves alongside multimodal UX data streams over space and time. While visualization techniques play a central role in supporting the analysis of such data, existing approaches typically scatter these individual dimensions across detached views, failing to examine how physiological and experiential signals relate directly to their embodied context. Immersive analytics opens perspectives by allowing dimensions to be displayed in context directly within the 3D environment in which the experience takes place. In this paper, we address how the combined visualization of human motion and multimodal UX data streams in 3D scenes contribute to immersive analytics. We structure our investigation around three core elements: data synchronization, body-linked immersive representation, and analytical perspectives. Specifically, we present an approach that synchronizes body kinematics within a 3D scene while establishing a direct data-to-body mapping that binds physiological and contextual UX signals directly onto moving 3D skeleton. To support discovery of complex behavioral patterns, we combine complementary analytical perspectives, including exocentric viewpoints for spatial overviews, egocentric viewpoints for situated inspection, and abstract temporal views for focused trend analysis. We demonstrate the analytical potential of our approach through a systematic heuristic walkthrough and task-oriented scenarios based on an existing human motion dataset. By bridging the spatial-physiological correlation gap, this work contributes an exploratory framework for investigating the dimensions of immersive systems and informs future research on visualization designs for examining embodied interaction in XR.