ShadAR: Real-time Prompting On-demand Visual Augmentation in Augmented Reality
TL;DR
ShadAR is introduced, an AR system that allows users to describe expected visual augmentations for changing their vision in real time, and can graphically manipulate AR passthrough in real time via shader injection.
Abstract
Augmented Reality can reshape our visual perception (e.g., simulating color blindness) yet such effects are typically pre-programmed by experts, making visual augmentation static and inflexible. We introduce ShadAR, an AR system that allows users to describe expected visual augmentations (e.g.,"Blur all people") for changing their vision in real time. ShadAR can graphically manipulate AR passthrough in real time via shader injection. When receiving a user's prompt, ShadAR uses LLMs (GPT5.2) to interpret the user's intent, analyzes the user's view with object detection (YOLOv26), generates a pixel shader, and applies it to the targeted region on AR passthrough. To understand usage patterns, we invited 21 HCI researchers to use ShadAR, identifying four application domains (design, health, productivity, and social interaction) and two prompt-framing patterns: procedural prompts specifying what to modify and how ("Highlight the book."), and end-goal prompts conveying desired mental or behavioral states ("Make me laugh."). Such end-goal prompts suggest a future in which users specify outcomes (e.g.,"Make me not miss my flight") while the system determines how to realize them, though often with inherently indeterminate results. We discuss design opportunities for ShadAR and ethical concerns.
