Skip to content
Open access

Beyond Conventional ASR Metrics: Evaluating Semantic Preservation in Low-Resource Speech Recognition

2026 · International Journal of Advanced Computer Science and Applications · 0 citations · 23 references

Abstract

Word error rate (WER) and character error rate (CER) quantify lexical discrepancies in automatic speech recognition (ASR) but do not directly indicate whether the intended message remains intact. This exploratory study compared lexical accuracy with human-rated preservation of essential meaning in Malay-English code-switched conversational speech. Four complete inference pipelines were evaluated on 30 purposively selected windows from one recording (826 reference words): Mesolitica Malaysian Whisper Large-v3-Turbo-v3 (Pipeline 1), OpenAI Whisper Large-v3-Turbo through faster-whisper (Pipeline 2), Qwen3-ASR-1.7B with a combined Malay-English contextual prompt (Pipeline 3), and the same Qwen3-ASR model without that prompt (Pipeline 4). Three evaluators independently assessed 120 reference-hypothesis pairs; a two-of-three majority constituted the semantic endpoint. Pipeline 4 had the lowest WER (29.18%) and CER (20.60%), whereas Pipeline 1 had the highest observed majority-rated semantic-preservation proportion (53.3%, versus 36.7% for Pipeline 4). The overall pipeline effect was significant for WER (Friedman chi-square (3) = 15.64, p = 0.0013) and CER (chi-square (3) = 18.17, p < 0.001), but not for semantic preservation (Cochran's Q = 5.57, p = 0.134); no paired semantic contrast remained significant after Holm adjustment. The Pipeline 1–Pipeline 4 semantic difference was 16.7 percentage points (paired-window bootstrap 95% confidence interval: −6.7 to 40.0 percentage points). Overall Fleiss' kappa was 0.529. Different descriptive rankings were therefore observed, but this single-recording sample does not establish significant differences in semantic performance or general model superiority. These findings support further evaluation of human semantic judgments alongside lexical metrics, with independent validation across recordings, speakers, and inference implementations.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.