EviSI, a large language model evaluation agent combining Multidimensional Quality Metrics (MQM) with criteria developed with professional interpreters, is proposed, showing positive system ranking correlations with COMET throughout.
Abstract
Low-latency simultaneous speech-to-speech translation must keep pace with ongoing speech while preserving key information. To meet these demands, systems use segmentation, reformulation and condensation to reorganize and rephrase information. However, metrics developed for text translation, including BLEU and COMET, may not consistently distinguish faithful adaptations from semantic errors. We propose EviSI, a large language model evaluation agent combining Multidimensional Quality Metrics (MQM) with criteria developed with professional interpreters. Shared source evidence guides assessment across four dimensions: Anchor, Event, Logic and Fluency. Verified errors are deduplicated before deterministic scoring. On human-rated English to Chinese and Chinese to English data, EviSI recovers the aggregate English to Chinese human system ranking. Mean within-dataset Kendall correlations for system rankings reach 0.707 and 0.467, respectively, exceeding evaluated BLEU and COMET baselines. A multilingual extension to five directions without human ratings retains the dimensions and scoring rule, showing positive system ranking correlations with COMET throughout.
This work proposes Agentic-GER, an LLM-based agent for terminology correction in long-form speech, which uses global context from the full transcript to identify suspicious terms and resolve ambiguous hypotheses.
Yan-Qiao Zhu, Wu-Peng Wang, Zhifu Gao et al.· 0 citations
This paper describes the system for Task~2 of the second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge, which adapt Qwen3-Omni-30B-A3B-Instruct with a segment-evidence-aware data and post-training pipeline and obtains 90.92% accuracy on the final official evaluation set.
H. Le, L. Nguyen, Minh Tri Dao· 2nd Workshop on Multilingual...· 1 citation
Reliability-Aware Adaptive Inference (RAAI), a training-free test-time framework that performs Expected Calibration Error-aware routing and prompt fusion, and uses a mid-layer Risk Index to gate sequential reasoning, allocating compute only when it is likely to help and suppressing harmful bilingual redundancy, is addr...
Ekata Mitra, Ameeta Agrawal· Conference on Empirical Meth...· 0 citations
This work introduces an alternative inspired by language-learning assessment, using an open-weight-LLM QA protocol that measures salient content preservation that aligns more closely with human rankings and is six to seven times more paraphrase-invariant than BLEU-4.
Oline Ranum, Edward Fish, Simon Hadfield et al.· 0 citations
MuLA-Bench exposes conditional failure patterns that a single long-context score does not capture, and evaluates ten audio-language models and conducts pooled diagnostics on a fixed eight-model cohort.
Ze-Yu Yang, Xin-Yu Zhang, Zi-Bo Bi et al.· arXiv.org· 2 citations
Voice agents depend on accurate automatic speech recognition (ASR) to act on what callers say, yet ASR is evaluated on read, English-centric speech with word error rate (WER), which penalizes surface rather than semantic differences. We introduce mu-bench, a dataset of 4,270 caller utterances from 250 phone calls to an...