This work analyzes over 790,000 translation outputs from 12 LLMs across 22 language pairs (LPs) and identifies 12 recurring noise patterns, which are group into formatting and content noise, and constructs TransClean, a controlled benchmark of 9,900 pairs of noisy and clean translation outputs.
Abstract
Large language models (LLMs) are increasingly used for machine translation, yet their outputs often contain additional text beyond the translation itself, such as language labels, explanations or bilingual repetitions, which we term translation noise. Despite its prevalence, this problem lacks dedicated benchmarks and systematic study. We analyze over 790,000 translation outputs from 12 LLMs across 22 language pairs (LPs) and identify 12 recurring noise patterns, which we group into formatting and content noise. Building on the observed patterns, we construct TransClean, a controlled benchmark of 9,900 pairs of noisy and clean translation outputs, comprising 8,800 synthetically generated instances and 1,100 manually curated authentic instances. We evaluate two extraction approaches on the TransClean benchmark: 1) a span-based extraction method leveraging translation quality estimation models for span detection, and 2) an LLM-based extraction method that prompts an LLM to isolate the translation. Our benchmark and analysis provide the first systematic framework to evaluate and improve the cleanliness of LLM translation outputs.
This work combines vocabulary surgery for tokenizer adaptation without full retraining, cross-lingual transfer of quality classifiers via translation, enabling filtering without target-language annotations, and generation of instruction data through translation, task conversion, and targeted synthesis.
Yurii Paniv, Bohdan Didenko, Mykola Haltiuk et al.· 4 citations
Fragment-Shot Prompting is introduced, a novel few-shot prompting method that aims to retrieve examples for every word occurring in the sentence to be translated, illustrating their use and meaning in context.
Samuel Frontull, Thomas Ströhle· International Conference on...· 1 citation
This paper proposes automated methods to construct high-quality WordNets using large language models (LLMs) to generate missing lemmas to address the synset shortfall in non-English and low-resource languages.
Johann Bergh, J. Waitelonis, Melanie Siegel· 0 citations
A controlled evaluation framework for multilingual natural language inference that stress-tests semantic consistency using synthetic logic-based premise–hypothesis pairs in both monolingual and code-switched conditions is presented.