This work instructed a voice-controlled GPT-4o LLM to send commands to an OpenSpace instance, effectively providing the LLM with the the ability to steer the visualization software as a museum facilitator would for educational shows.
Can interactive vision-and-language agents learn not just what to say but also when to say it? Current language models rarely plan over whether and when to realize a real-time response to a user. However, providing accurate and timely support for human decision-making, such as when guiding visually impaired individuals...
Large language and multimodal models have been largely employed to reactively generate content and assistance. In contrast, we explore how models can be employed to provide proactive assistance that fluidly supports ongoing human activities. We identify and tackle challenges in such streaming scenarios—where models mus...
Sean Andrist, Dan Bohus, Maia Stiber et al.· Proceedings of the 28th Inte...· 0 citations
People communicate how things should move by combining words with demonstrations:"open it like this."We present an interaction technique that brings this expressive resource to conversational 3D authoring. Building on"Put-That-There,"our system combines speech, pointing, and spatiotemporal demonstrations to specify edi...
Hamza El Alaoui, Jeffrey P. Bigham, Jun Rekimoto· 0 citations
MUSE is presented, an interactive meta-agent that enhances user understanding and control of agentic data science systems by dynamically restructuring low-level execution traces into multiple semantic levels that support navigation from high-level overviews to low-level implementation details.
Wei-Hao Chen, Weixi Tong, Yuan Tian et al.· 0 citations
AI-powered gameplay support agents hold promise for game-based learning, yet grounding generative models in structured game data remains an open challenge. We present PEARL (Parallel Education Agent for Reflection and Learning), a dual-component Retrieval-Augmented Generation (RAG) system that combines semantic knowled...
Jia-Hong Li, Sai Siddartha Maram, Atieh Kashani et al.· arXiv.org· 0 citations
Forest Classroom is a mobile, spatially situated conversational Augmented Reality (AR) system designed to support help-seeking during learning activities. Learners interact by voice with an embodied avatar presented within an immersive three-dimensional environment. Rather than acting as an authoritative tutor, the ava...
Manjeet Singh, Sajjanhar Atul, Yi Wang· Proceedings of the 2026 ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.