Skip to content

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

Sep 2026 · arXiv.org · Vol abs/2609.21663 · 0 citations · 18 references
Computer Science

TL;DR

The results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.

Abstract

Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.

View source

Similar papers

Open access 2026

Beyond Conventional ASR Metrics: Evaluating Semantic Preservation in Low-Resource Speech Recognition

Word error rate (WER) and character error rate (CER) quantify lexical discrepancies in automatic speech recognition (ASR) but do not directly indicate whether the intended message remains intact. This exploratory study compared lexical accuracy with human-rated preservation of essential meaning in Malay-English code-sw...

Nurul Emilia Diyana Abdul Malik, M. I. Mohd Tamrin, Airul Azizan Zainudin et al. · 0 citations
Open access Oct 2026

Semantic Segment Alignment and Granular ASR Performance Evaluation with Segment- and Word-Level WER

Word Error Rate (WER) is the de facto standard for evaluating automatic speech recognition (ASR) systems. Yet, conventional WER is a global average, which obscures critical performance variations across semantically important transcript segments; segment-level WER can reveal deeper insights for performance evaluation o...

Jian-Hua Liu · 0 citations
Conference Open access 2026

Beyond Rankings: Insights from ALTAE’s PTT-MULTITAN Submission to the WMT 2026 Shared Task on Automated Translation Quality Evaluation Systems

We describe PTT - MULTITAN , our submission to the WMT26 MT Evaluation shared task, Task 3 (error-free segment detection). The task requires a binary decision – whether a machine translation (MT) contains any error – without access to reference translations, and is scored using the Matthews Correlation Coefficient (MCC)...

Li-Chao Zhu, Maria Zimina, Nicolas Froeliger · 0 citations
#natural language process... Preprint Oct 2026

Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. The...

Anubhav Jangra, Adrian de Wynter · 0 citations
Conference Open access 2026

SpinPop: A Fast Spin Metric for the WMT26 Metrics Shared Task

We describe our submission to the WMT26 Shared Task on Automated Translation Quality Evaluation. Our primary system is a per-language-pair, z -normalized, weight-free ensemble that combines six complementary signals: our novel metric SpinPop, COMET-22, and four large language model judges. SpinPop is training-free, con...

Jian-Jun Xu · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.