Skip to content
Conference Open access

StellAR: A Multimodal Generative AI Framework for Context-Aware Augmented Reality in Stem Education

Aug 2026 · International Conference on Computing Communication Control and automation · pp. 1-8 · 0 citations · 41 references

TL;DR

StellAR is proof that generative AI can become the solution that eliminates the content creation barrier to scale in AR for STEM learning and is proof that generative AI can become the solution that eliminates the content creation barrier to scale in AR for STEM learning.

Abstract

Although Augmented Reality (AR) has shown great promise for use in education, the reliance on 3D modeling expertise to develop AR educational content has hindered its widespread adoption and implementation in STEM, or science, technology, engineering, and mathematics classrooms. This paper introduces a context-aware immersive learning framework, called StellAR, which automatically transforms static educational PDF files and textbooks into interactive AR learning sessions, without any programming or 3D modeling skills from educators. At the heart of StellAR is a Multimodal Retrieval-Augmented Generation (RAG) pipeline driven by Gemini 1.5 Pro, which validates semantic domains and extracts keywords from uploaded documents. A hybrid Asset pipeline that routes the requests according to the cosine similarity between the concepts, and fetches pre-compiled GLB assets from a vector store (cos 0.85) otherwise fetches a new concept through a generative path using 2D reference images and a ShapeVAE-based implicit Signed Distance Function (SDF) representation using Hunyuan3D 2.1, a flow-matching Diffusion Transformer for geometry-consistent 3D model generation. An Intelligent Tutoring System (ITS) is embedded in the system and provides adaptive audio explanations in real time, starting right at the time of context extraction, even though 3D assets are loaded asynchronously - making passive AR viewing into active guided problem-solving. With a “Custom Classroom” module, teachers can select AR lessons without technical expertise. The system has been tested on the retrieval path with 50 retrieval tests and has achieved an AR plane detection accuracy of 93.2%, a semantic analysis latency of below 200 ms and Time-to-Interaction of less than 2.2 seconds. StellAR is proof that generative AI can become the solution that eliminates the content creation barrier to scale in AR for STEM learning.

Read PDF

Similar papers

#generative ai Open access Nov 2026

H-GIM: a human-centered generative immersive multimedia framework for adaptive immersive multimedia systems

A synthetic data generation pipeline that automatically produces high-fidelity 3D assets using procedurally guided generative models and demonstrates the potential of generative and multimodal AI to enable scalable, adaptive, and human-centered immersive multimedia systems.

Imen Chebbi, S. Abidi, Leila Ben Ayed · 0 citations
Conference Open access 2026

Framework Digital History Textbook Assisted Augmented Reality and Artificial Intelligence to Enhance High School Students’ Historical Literacy

This study aims to develop a conceptual framework for a digital history textbook assisted by Augmented Reality (AR) and Artificial Intelligence (AI) to enhance high school students’ historical literacy. The research employs a qualitative descriptive approach through a literature review and conceptual analysis of studie...

Wawan Darmawan, Tarunasena, D. Pangestu · 0 citations
Conference Open access Sep 2026

AI-Assisted Scene Synthesis in the 3D Educational Game Authoring Tool GameTULearn

An AI-assisted scene-synthesis workflow that translates prompt-based natural-language descriptions into editable 3D room layouts inside GameTULearn, which supports a hybrid authoring model in which AI accelerates scene creation, but users remain responsible for review and refinement.

Florian Horn, Alessandro Vincenti, S. Göbel · 0 citations
Conference Open access Oct 2026

Integrating AI-Generated 3D/360° Content into VR-Based Secondary Education

This paper presents a pedagogical framework for integrating AI-generated educational content into Virtual Reality (VR)-based teaching within SmartLab-supported secondary education. The framework addresses the limited classroom adoption of AI and immersive technologies by combining AI-assisted content generation, VR int...

Vasile-Ioan-Grigore Avram, Ovidiu Cosma, N. Barkoczi · 0 citations
Open access Sep 2026

Design an Immersive Learning Media Based on Spatial Augmented Reality Using Hand Tracking in the ChemMaster Application

Today’s technological progress gives educators new ways to create engaging and effective learning experiences. This research introduces ChemMaster, a learning media application that combines Spatial Augmented Reality (AR) and hand-tracking technology to help high school students understand complex chemistry concepts. T...

Rigel Tandilolo, Maria Angelica Vinesytha Chandrawan, Alicia Juanita Lisal et al. · 0 citations
Review Open access Sep 2026

AI-Enhanced Augmented Reality Framework for Interactive Learning in Higher Education

Augmented Reality (AR) is increasingly used in higher education to improve visualization, engagement, and practice-based learning, yet existing studies often discuss learning benefits, implementation patterns, and adoption barriers separately and provide limited guidance for AI-supported personalization. This study con...

Amir ’Aatieff Amir Hussin, A. A. Puzi, Siti Nur Raihannah Nazrul et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.