The results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.
Abstract
Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.
Word error rate (WER) and character error rate (CER) quantify lexical discrepancies in automatic speech recognition (ASR) but do not directly indicate whether the intended message remains intact. This exploratory study compared lexical accuracy with human-rated preservation of essential meaning in Malay-English code-sw...
Nurul Emilia Diyana Abdul Malik, M. I. Mohd Tamrin, Airul Azizan Zainudin et al.· International Journal of Adv...· 0 citations
Word Error Rate (WER) is the de facto standard for evaluating automatic speech recognition (ASR) systems. Yet, conventional WER is a global average, which obscures critical performance variations across semantically important transcript segments; segment-level WER can reveal deeper insights for performance evaluation o...
We describe PTT - MULTITAN , our submission to the WMT26 MT Evaluation shared task, Task 3 (error-free segment detection). The task requires a binary decision – whether a machine translation (MT) contains any error – without access to reference translations, and is scored using the Matthews Correlation Coefficient (MCC)...
Li-Chao Zhu, Maria Zimina, Nicolas Froeliger· Proceedings of the Eleventh...· 0 citations
Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. The...
We describe our submission to the WMT26 Shared Task on Automated Translation Quality Evaluation. Our primary system is a per-language-pair, z -normalized, weight-free ensemble that combines six complementary signals: our novel metric SpinPop, COMET-22, and four large language model judges. SpinPop is training-free, con...
Jian-Jun Xu· Proceedings of the Eleventh...· 0 citations