A multilevel computing framework for behavior recognition and interaction in mixed reality workspaces

Abstract

Mixed Reality (MR) technology, as an emerging interactive medium, has seen widespread application in various everyday fields in recent years, offering significant convenience for intelligent interactions between humans and spaces. However, current MR environments primarily rely on single-input interaction methods, where specific behaviors or body movements are recognized to drive changes in the virtual environment. Such approaches often overlook the comprehensive and interrelated nature of human behaviors, potentially diminishing the immersive user experience. This paper proposes an improved interaction model that aims to comprehensively perceive multi-scale human behaviors, including social, global-postural, local-gestural, and facial scales. Using a smart office space as the design prototype, the system integrates body tracking and affective computing techniques, and employs Grasshopper and Unity for real-time spatial modeling and rendering. This allows the MR environment to dynamically respond to user behaviors and support flexible spatial transformations. An experimental study provides initial evidence that the proposed framework can effectively sense user behaviors and dynamically adapt environmental changes. The results suggest potential improvements in spatial immersion and interaction naturalness, providing a reference for further exploration of behavior-driven interaction in intelligent MR environments.