A systematic empirical analysis demonstrates that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates, and proposes CuRIL, a reinforcement learning framework that internalizes cultural reasoning.
Abstract
Automatic translation quality metrics trained on general-domain corpora systematically fail on social media content, where communicative intent is encoded in culturally loaded expressions (internet slang, homophonic ciphers, and platform-specific idioms) rather than surface token patterns. We conduct a systematic empirical analysis demonstrating that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates. We further show that this failure extends to large language model judges: Qwen3-235B achieves Cohen's kappa of only 0.162, revealing that the bottleneck is not reasoning capacity but cultural grounding: models lack the domain-specific cultural knowledge needed to identify which aspects of a translation require scrutiny. To address this, we propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning: cultural annotations are prepended inside the model's reasoning, excluded from policy gradients via a token-level loss mask, and injected with a probability that decays to zero over training, progressively forcing autonomous cultural judgment. On a 1,444-sample human-annotated social media translation benchmark, Qwen3-8B trained with CuRIL achieves Cohen's kappa 0.370 and Exact Match accuracy of 45.22%, approaching Gemini-3.1-Pro with 30x fewer parameters and surpassing models up to 235B in scale. We further demonstrate that our judge produces reliable reward signals for downstream translation optimization, reducing the low-quality translation rate by over 20 percentage points under independent human evaluation.
It is argued that collapsing cultural competence into a single scalar incentivizes "probabilistic flattening," and that recognizing the unalignable class is a precondition for AI that respects, rather than erases, cultural divergence.
This study evaluates the performance of three machine translation tools: ChatGPT (OpenAI GPT-3.5, March 2024), Google Translate (web version, April 2024), and DeepL (free web version, April 2024), focusing on a corpus of Algerian Supreme Court judgments.
Mohamed Hamza Merabet, Nicolás Roser Nebot· Comparative Legilinguistics· 0 citations
A benchmark for evaluating whether LLMs can recover situated pragmatic meanings in Chinese online comments is introduced, and case analysis shows that models often recognize broad irony or playfulness while misidentifying the mechanism or interactional move.
Yin-Hong Shi, Jun-Jie Ma, Emma Jiren Wang et al.· 0 citations
This work argues that literary interpretation offers a useful setting for studying cultural referencing and reuse: how texts invoke, repeat, and transform earlier expressions across historical and linguistic contexts and proposes linking evidence-centered benchmarks, evaluation that preserves scholarly disagreement, an...
Daniel Hershcovich, Alexander Conroy, Jens Bjerring-Hansen· arXiv.org· 0 citations
Results show that instruction-tuned models like GPT-4o and Gemini 1.5 Pro outperform smaller models in both automatic and human evaluations, and LLM-as-a-Judge evaluation correlates strongly with human assessment.
Nasser Thmer, Ali Allaith, Muhammad Shoaib· International Conference on...· 0 citations
Real-world communication often requires pragmatic reasoning: interpreting meanings implied through context and cultural convention rather than stated literally. Existing pragmatic evaluation remains largely limited to English and high-resource languages, leaving Indic languages unexplored despite their linguistic and c...
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.