Beyond Conventional ASR Metrics: Evaluating Semantic Preservation in Low-Resource Speech Recognition
Abstract
Word error rate (WER) and character error rate (CER) quantify lexical discrepancies in automatic speech recognition (ASR) but do not directly indicate whether the intended message remains intact. This exploratory study compared lexical accuracy with human-rated preservation of essential meaning in Malay-English code-switched conversational speech. Four complete inference pipelines were evaluated on 30 purposively selected windows from one recording (826 reference words): Mesolitica Malaysian Whisper Large-v3-Turbo-v3 (Pipeline 1), OpenAI Whisper Large-v3-Turbo through faster-whisper (Pipeline 2), Qwen3-ASR-1.7B with a combined Malay-English contextual prompt (Pipeline 3), and the same Qwen3-ASR model without that prompt (Pipeline 4). Three evaluators independently assessed 120 reference-hypothesis pairs; a two-of-three majority constituted the semantic endpoint. Pipeline 4 had the lowest WER (29.18%) and CER (20.60%), whereas Pipeline 1 had the highest observed majority-rated semantic-preservation proportion (53.3%, versus 36.7% for Pipeline 4). The overall pipeline effect was significant for WER (Friedman chi-square (3) = 15.64, p = 0.0013) and CER (chi-square (3) = 18.17, p < 0.001), but not for semantic preservation (Cochran's Q = 5.57, p = 0.134); no paired semantic contrast remained significant after Holm adjustment. The Pipeline 1–Pipeline 4 semantic difference was 16.7 percentage points (paired-window bootstrap 95% confidence interval: −6.7 to 40.0 percentage points). Overall Fleiss' kappa was 0.529. Different descriptive rankings were therefore observed, but this single-recording sample does not establish significant differences in semantic performance or general model superiority. These findings support further evaluation of human semantic judgments alongside lexical metrics, with independent validation across recordings, speakers, and inference implementations.