Sep 2026· Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies· Vol 10, pp. 1 - 25· 0 citations· 73 references
TL;DR
This work introduces LEMuR, a language embedded 3D segmentation and tracking pipeline that enables real-time sharing and synchronization of everyday physical objects in MR and examines how LEMuR supports object-mediated gestures, proxemics, and artifact cues, revealing emergent interaction patterns unique to hybrid physical-digital collaboration.
Abstract
Mixed reality enables distributed collaboration across distinct physical spaces. However, collaborator interactions are restricted from shared manipulation of physical objects in their surroundings to aid communication. Existing approaches either substitute low-fidelity virtual proxies for physical objects, stripping away the communicative context essential for interpreting actions, or rely on external cameras that constrains interaction to fixed capture regions and impede practical deployment. We introduce LEMuR, a language embedded 3D segmentation and tracking pipeline that, following an offline static scene reconstruction phase, enables real-time sharing and synchronization of everyday physical objects in MR. Users specify objects to share through natural language queries, then LEMuR selectively visualizes it and synchronizes local manipulations as high-fidelity reconstructions for remote collaborators. Through a user study with 12 participants grounded in Knapp's framework of nonverbal communication, we examined how LEMuR supports object-mediated gestures, proxemics, and artifact cues, revealing emergent interaction patterns unique to hybrid physical-digital collaboration. We further evaluate pipeline performance and discuss design implications for object-centred multi-user MR systems.
STACK is presented, to the knowledge the first benchmark pairing real multi-party construction video with dense 3D annotations and goal-paired judgment, exposing a gap between perception and judgment in current VLMs and informing their use as analyzers of collaborative group dynamics.
Changsoo Jung, Sheikh Mannan, Jack Fitzgerald et al.· Proceedings of the 28th Inte...· 0 citations
Joint 4D reconstruction of a human and object from a monocular video is essential for understanding how people interact with their environment. While recent methods have made noticeable progress, capturing such human-object interactions remains difficult due to two main challenges. First, a video typically contains com...
Hyeongjin Nam, T. Lee, Daniel Sungho Jung et al.· 0 citations
We propose Tetris3D, a generative framework for single-image 3D scene reconstruction that recovers objects which are physically and geometrically coherent as a scene. Existing methods often generate objects independently or couple them implicitly, providing limited guidance for ensuring fine-grained spatial compatibili...
Jaeyeon Kim, Ji-Ze Jang, Jongmin Lee et al.· 0 citations
Mixed Reality (MR) expands interaction beyond traditional screens by embedding virtual interfaces within the physical environment. This affordance makes MR a powerful medium for collaboration, as users can jointly reference and coordinate around shared virtual interfaces placed in the real world. However, most collabor...
Khadija Arif, Talha Khan· Proceedings of the 2026 Comp...· 0 citations
Remote collaboration can greatly benefit from the improved reconstruction and tracking capabilities of optical see-through head-mounted displays. A remote helper can efficiently assist a local worker and convey complex instructions by placing 3D instructions into the environment. If a high quality reconstruction is not...
Michael Anderson, Dieter Schmalstieg, Alexander Plopski· Proceedings of the 2026 ACM...· 0 citations
Future immersive computing systems must understand physical environments not only within the current field of view, but persistently, as those environments change across repeated visits. Recent functional 3D scene graphs produce semantically rich representations of objects, affordances, articulated parts, and functiona...
Song Tu Linh Do, Mehruba Tanzid Ontika, Dhara Desai et al.· Proceedings of the 4th ACM W...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.