Algorithmic analysis of transcription errors in human and Whisper speech recognition*

Abstract

Previous studies have evaluated the speech recognition performance of human listeners and automatic speech recognition (ASR) systems using aggregate measures such as word error rate; however, these measures provide limited information about the nature of transcription errors. The current study introduces an algorithmic framework for transcription error analysis, focusing on substitution errors in content words. For each substitution pair, we calculated continuous measures of phonological and semantic similarity between the error and the target word. We examined these measures in transcriptions produced by human listeners and the Whisper ASR system during speech-in-noise recognition of first-language (L1) and second-language (L2) English speech. Both phonological and semantic similarity increased with the signal-to-noise ratio (SNR), with a descriptively larger increase in phonological similarity. We found no significant evidence that these SNR-related changes differed between humans and Whisper or between L1 and L2 speech. In an exploratory analysis, substitution errors were classified into four categories: morphological errors (e.g., tell vs. tells), semantic substitutions (e.g., large vs. big), phonetic confusions (e.g., cap vs. cat), and random-like guessing (e.g., school vs. open). As SNR increased, random-like guessing decreased, whereas the combined proportion of phonetic confusions and morphological errors increased. These findings demonstrate that similarity-based error analysis provides insights into the nature of transcription errors beyond aggregate metrics.