Skip to content

Demo abstract: An XR Platform that Integrates Large Language Models with the Physical World

· 0 citations · 5 references

TL;DR

XaiR is a system that integrates MLLMs with XR to enable AI-driven spatial reasoning and interaction and employs a client-server architecture in which an XR headset (client) captures spatial data, generates 2D snapshots of the 3D environment, and renders augmented reality (AR) content.

View source

Similar papers

Sep 2026

ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptions

A unified framework for generating physically consistent dynamic 3D scenes from text, with outputs directly executable in Unreal Engine, using the ESG, which specifies entities with physical attributes, spatial relations, and event-driven timelines in a machine-checkable form.

Xin Fang, Zhi-Yuan Fang, Ren-Gan Xie et al. · 0 citations
Preprint Oct 2026

BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References

BlenderWorld is introduced, the first comprehensive benchmark dedicated to evaluating frontier models on realistic 3D creation tasks, and reveals a critical bottleneck: while current models can parse broad instructions, they still profoundly struggle with fine-grained 3D spatial coherence and accurate structural reprod...

Xinyu Fang, Yu-Chi Wang, Wei-Hao Yu et al. · 1 citation
Preprint Oct 2026

NeoWorld-3: From Pixels to Code-Defined Physical Worlds

Simulation-ready world reconstruction demands physical interactivity beyond visual realism, which existing black-box 3D generation fails to deliver. We present NeoWorld-3, a coding-agent framework that reconstructs interactive 3D worlds from visual observations via tunable, executable programs. At its core, NeoSDK prov...

Eachen Soong, Zi-Jian Niu, Hao Xu et al. · 1 citation
Book Open access Nov 2026

ArtiWorld: LLM-Driven Articulation of 3D Objects in Scenes

Building interactive simulators and scalable robot-learning environments requires a large number of articulated assets. However, most existing 3D assets used in simulation are rigid, and manually converting them into articulated objects with executable kinematic structures is highly labor-intensive, time-consuming, and...

Yi-Xuan Yang, Lu-Yang Xie, Zhen Luo et al. · 0 citations
Preprint Oct 2026

Code2Worlds-2: Physically Plausible Agent Harness for Embodied World Simulation

Recent coding large language models (LLMs) can synthesize dynamic 4D worlds from natural language, but the resulting content remains a visual artifact. It lacks physical semantics such as the collision geometry, mass, friction, inertia, and joint structure, required to enable it to simulate the physical world, and its...

Hao-Yu Zhang, Si-Ming Li, Ze-Yu Zhang et al. · 1 citation
#artificial intelligence Preprint Oct 2026

RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work

Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability is to understand what the worker is doing, seeing through the human's camera or smart glasses, and assist them in documentation. Despite advances in vision language models, the...

Andrew Perrault, Anmol Takiar, Dominik Winecki et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.