Did You Hear That? Detecting AI Voice Agents by Exploiting Their Robustness

Abstract

AI voice agents enable scalable vishing and impersonation attacks, yet existing detection methods face practical limitations. Artifact-based detectors degrade as synthesis quality improves, challenge–response tests are socially awkward, and adversarial jamming must often be loud or continuous to disrupt modern automatic speech recognition (ASR). We introduce HEARD, a detection framework built on a new principle: instead of perturbing the AI while remaining imperceptible to humans, HEARD applies brief, conspicuous distortions that humans notice but robust ASR systems largely ignore. A human caller is likely to comment on the disturbance, misunderstand the utterance, or request repetition, whereas an AI agent continues coherently from the recovered transcription. We map fifteen audible distortions into three families and evaluate them across six ASR models spanning multiple architectures and five years of development, as well as an experiment involving 41 human volunteers. Using semantic similarity and an LLM-based conversational-legibility metric, we find that across 2,421 LibriSpeech and TED-LIUM utterances, modern ASRs preserve a mean legibility of 97.8%, while the best distortion keeps every model at or above 98.3%. Moreover, performance improves with model recency, making stronger ASR systems easier, not harder, to distinguish from humans.