This work argues that one meaningful and currently unmeasured axis is referential reach, the distance a segment must look back into its document to resolve the entities and pronouns it contains, and formalizes this as discourse dependency (DDP), a metric-free, source-side measure computed from named entity re-mentions and pronominal coreference.
Abstract
Recent calls for harder machine translation benchmarks have not clarified what difficulty should mean. We argue that one meaningful and currently unmeasured axis is referential reach, the distance a segment must look back into its document to resolve the entities and pronouns it contains. We formalize this as discourse dependency (DDP), a metric-free, source-side measure computed from named entity re-mentions and pronominal coreference. Validated against gold coreference, DDP errs one-sidedly in 99.2% of segments, so a high-DDP segment is certified to require long-range context. Applying DDP to WMT24++ and WMT25 shows that both are heavily skewed toward low-DDP segments, which domain labels do not distinguish. Building on DDP, we compare five context injection strategies in an English-Korean post-editing setup, varying context size and selection. As DDP grows, no strategy keeps pace with human post-editing. On segments with DDP>= 15 raters prefer human translations, while automatic metrics register no difference. As frontier systems saturate aggregate scores, DDP shifts evaluation from how well models score to how far they can reach.
TermJudge is introduced, a document-level terminology metric that assigns an interpretable verdict to every term occurrence, and it is observed that glossary injection improves terminology translation in all paired comparisons, by removing genuine errors rather than valid variation.
Nicolas M. Dahan, François Yvon, Rachel Bawden· Proceedings of the Eleventh...· 0 citations
J-PragEval-v0 is introduced, a minimal-pair benchmark isolating four such phenomena from surface fluency, and Pragmatic Representation Steering is specified, a parameter-free inference-time method that edits residual-stream activations along the class-mean-difference directions probing identifies.
The main contribution is a reusable evaluation method for historical language generation, together with a single interpretable score that subsumes the partial metrics without hiding them.
Javier Martín Arista, Matías Núñez· Applied Informatics· 0 citations
Fragment-Shot Prompting is introduced, a novel few-shot prompting method that aims to retrieve examples for every word occurring in the sentence to be translated, illustrating their use and meaning in context.
Samuel Frontull, Thomas Ströhle· International Conference on...· 1 citation
Natural Language Inference (NLI) processes pairs of sentences to extract their semantic relations. NLI has been a hot research topic, integrated as a main component in other NLP applications, from morphological spelling correction tasks to higher-level tasks such as machine translation and information extraction. Despi...