Be My Ears: Enabling Semantic Alerting for Earphone Wearers with Pheme

Abstract

Active Noise Cancellation (ANC) earphones provide immersive listening experiences but induce auditory isolation to earphone users, leaving them vulnerable to hazards and socially disconnected. We present Pheme, an agentic framework that transforms ANC earphones into context-aware auditory assistants. Informed by two user studies (N = 385 and N = 34) that reveal a high prevalence of missed critical sounds and significant linguistic ambiguity (73.6%) in user requests, Pheme employs a three-stage pipeline: (1) an LLM-based Query Understanding module to resolve intent and map ambiguous commands to acoustic targets; (2) Text-conditioned Audio Separation (via SAM Audio) to isolate target sounds from ambient noise; and (3) LALM-based Verification to validate detections against user intent, reducing false interruptions. We evaluate Pheme against a baseline that directly feeds ambient audio into an audio language model for target event detection, using 1,584 audio samples across seven real-world scenarios. Our results show that Pheme achieves 82.2% detection accuracy for user-specified audio events at + 5 dB SNR setup, outperforming the baseline by 9.5% while reducing the false positive rate by 11.9%. Moreover, Pheme significantly reduces operational overhead, cutting API calls by 48.1% and LALM audio processing workload by 67.3%.