SonicAR: Augmenting Human Hearing through AR Glasses

Abstract

Visualizing and interacting with the sounding objects is an important capability for AR glasses, with applications in accessibility, maintenance, and safety awareness. Marking where a sound comes from in the surrounding scene is the key to such interaction. Existing on-glasses interfaces use the small array on the frame to recover a coarse relative direction, without range or a location in the scene. Synthetic-aperture methods do localize sources accurately, but they run on dedicated platforms and offer little interaction to a wearer. We present SonicAR, a system that localizes sounds in the surrounding scene, visualizes them as AR content, and supports intent-driven interaction with them. SonicAR uses the SLAM tracking and scene reconstruction to fuse the multichannel audio on the glasses with the poses along the trajectory, accumulating acoustic evidence over the reconstructed scene. The wearer states which sound to seek or attend to, and a semantic model resolves this request against the sources SonicAR has found, which are then rendered in view. Further interaction lets the wearer adjust how the sound field is visualized, so that walking up to a source both inspects it and sharpens its estimate.