Skip to content
Open access

LEMuR: Language Embedded 3D Segmentation and Object Tracking for Mixed Reality

Sep 2026 · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · Vol 10, pp. 1 - 25 · 0 citations · 73 references

TL;DR

This work introduces LEMuR, a language embedded 3D segmentation and tracking pipeline that enables real-time sharing and synchronization of everyday physical objects in MR and examines how LEMuR supports object-mediated gestures, proxemics, and artifact cues, revealing emergent interaction patterns unique to hybrid physical-digital collaboration.

Abstract

Mixed reality enables distributed collaboration across distinct physical spaces. However, collaborator interactions are restricted from shared manipulation of physical objects in their surroundings to aid communication. Existing approaches either substitute low-fidelity virtual proxies for physical objects, stripping away the communicative context essential for interpreting actions, or rely on external cameras that constrains interaction to fixed capture regions and impede practical deployment. We introduce LEMuR, a language embedded 3D segmentation and tracking pipeline that, following an offline static scene reconstruction phase, enables real-time sharing and synchronization of everyday physical objects in MR. Users specify objects to share through natural language queries, then LEMuR selectively visualizes it and synchronizes local manipulations as high-fidelity reconstructions for remote collaborators. Through a user study with 12 participants grounded in Knapp's framework of nonverbal communication, we examined how LEMuR supports object-mediated gestures, proxemics, and artifact cues, revealing emergent interaction patterns unique to hybrid physical-digital collaboration. We further evaluate pipeline performance and discuss design implications for object-centred multi-user MR systems.

Read PDF

Similar papers

#small language model Book Open access Oct 2026

STACK: 3D Spatial Task Analysis in Collaborative Construction Keyframes

STACK is presented, to the knowledge the first benchmark pairing real multi-party construction video with dense 3D annotations and goal-paired judgment, exposing a gap between perception and judgment in current VLMs and informing their use as analyzers of collaborative group dynamics.

Changsoo Jung, Sheikh Mannan, Jack Fitzgerald et al. · 0 citations
Preprint Oct 2026

HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering

Joint 4D reconstruction of a human and object from a monocular video is essential for understanding how people interact with their environment. While recent methods have made noticeable progress, capturing such human-object interactions remains difficult due to two main challenges. First, a video typically contains com...

Hyeongjin Nam, T. Lee, Daniel Sungho Jung et al. · 0 citations
Preprint Oct 2026

Tetris3D: 3D Scene Generation With Objects That Fit Together

We propose Tetris3D, a generative framework for single-image 3D scene reconstruction that recovers objects which are physically and geometrically coherent as a scene. Existing methods often generate objects independently or couple them implicitly, providing limited guidance for ensuring fine-grained spatial compatibili...

Jaeyeon Kim, Ji-Ze Jang, Jongmin Lee et al. · 0 citations
Book Open access Oct 2026

CommonGround: Externalizing Mixed Reality Interfaces for Active Bystander Collaboration

Mixed Reality (MR) expands interaction beyond traditional screens by embedding virtual interfaces within the physical environment. This affordance makes MR a powerful medium for collaboration, as users can jointly reference and coordinate around shared virtual interfaces placed in the real world. However, most collabor...

Khadija Arif, Talha Khan · 0 citations
Book Open access Oct 2026

More does not Mean Better: Effects of Spatial Snapshot Annotations on Remote Collaboration

Remote collaboration can greatly benefit from the improved reconstruction and tracking capabilities of optical see-through head-mounted displays. A remote helper can efficiently assist a local worker and convey complex instructions by placing 3D instructions into the environment. If a high quality reconstruction is not...

Michael Anderson, Dieter Schmalstieg, Alexander Plopski · 0 citations
Book Open access Oct 2026

Towards Persistent Dynamic Scene Understanding for Immersive Computing

Future immersive computing systems must understand physical environments not only within the current field of view, but persistently, as those environments change across repeated visits. Recent functional 3D scene graphs produce semantically rich representations of objects, affordances, articulated parts, and functiona...

Song Tu Linh Do, Mehruba Tanzid Ontika, Dhara Desai et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.