Skip to content

The Blindness of Document-Level Translation Evaluation

Sep 2026 · 0 citations · 56 references
Computer Science

TL;DR

A counterfactual condition (MIX) is test with a counterfactual condition (MIX) in which each document combines segments drawn from different systems, preserving document-level presentation while breaking cross-segment consistency.

Abstract

Document-level machine translation (MT) evaluation extends segment-level protocols by presenting full documents to annotators, on the assumption that such presentation elicits document-level judgments. We test this assumption with a counterfactual condition (MIX) in which each document combines segments drawn from different systems, preserving document-level presentation while breaking cross-segment consistency. Across 18,420 expert Englis-to-Korean annotations and 14 automatic metrics, scores, system rankings, and error annotations are statistically equivalent between coherent and incoherent documents. Perception does not explain this: shown matched passages, raters identify the coherent one as the work of a single translator in 87.3% of trials. Document presentation does change how annotators work, but that change does not reach the recorded output. What is blind is the protocol, not the annotator. The concern is not that scores fall short, but that the resources invested in document-level systems, metrics, and annotation may not be measuring what they are intended to measure.

View source

Similar papers

#artificial intelligence Conference Open access Sep 2026

TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation

TermJudge is introduced, a document-level terminology metric that assigns an interpretable verdict to every term occurrence, and it is observed that glossary injection improves terminology translation in all paired comparisons, by removing genuine errors rather than valid variation.

Nicolas M. Dahan, François Yvon, Rachel Bawden · 0 citations
Preprint Aug 2026

STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translation

This work proposes STAR-masked Preference Optimization (StarPO), a framework that ranks document-level hypotheses by structural quality and utilizes a dynamic alignment mask to focus optimization on misaligned segments and demonstrates that StarPO significantly enhances translation quality and structural integrity.

Yichen Dong, Hao Wang, Junhui Li et al. · 1 citation
#natural language process... Open access Oct 2026

Document-Level Text Simplification in Estonian Using Large Language Models

Document-level text simplification involves transformations that go beyond sentence-internal edits, addressing discourse coherence, anaphora resolution, and cross-paragraph consistency. Despite advances in sentence-level simplification for high-resource languages, document-level simplification in morphologically rich,...

Meeri-Ly Muru, Eduard Barbu · 0 citations
#natural language process... Preprint Sep 2026

Improving Term Evaluation in Machine Translation: Variation Matters

It is argued for variation-aware evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation, and constraining MT with a glossary improves accuracy and consistency but degrades CTV by suppressing valid variation.

Nicolas M. Dahan, Ziqian Peng, François Yvon et al. · 1 citation · ⚡1
#natural language process... Preprint Sep 2026

FACET at WMT 2026 Automated Translation Quality Evaluation Task

FACET, the authors' reference-free submission to the WMT26 Automated Translation Quality Evaluation Task, is presented, which decomposes evaluation into Fluency, Accuracy, and Consistency passes and gives each pass only the context its error type requires.

Ahrii Kim, Chanjun Park, Seong-heum Kim · 0 citations

Related blog posts

Microsoft Research Blog Jul 8, 2026

Flint: A visualization language for the AI era

Short chart specifications are easy to write, but often produce uninspiring results. Flint is an open-source visualization language that offers a middle path, letting AI agents create expressive charts from compact, human-editable specifications. The post Flint: A visualization language for the AI era appeared first on Microsoft Research.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.