Skip to content

Fair Fact-Checking: Closing the Cross-Lingual Gap in LLM Factual Judgement with RoSh

Sep 2026 · 0 citations · 25 references
Computer Science

TL;DR

RoSh is proposed, a per-language shift and rotation of the residual stream, computed in closed form at three layers, with no training and no weight modified, that improves every model and closes 75% of the gap on average, helping most where the model was worst.

Abstract

Misinformation on social media remains a critical problem, and more and more people settle it by asking a language model instead of a fact checker. Whether models judge such claims reliably is debated; whether they judge them equally well in every language people ask in has gone almost unasked. We test eight models from five families, 3B to 70B, on 1,500 encyclopedic factual claims that exist in identical form in eight languages. English is judged better than every other language on every model, and the gap is widest on the smallest ones, where Llama-3B on Arabic is no better than guessing. Existing remedies retrain on more multilingual data or fit an unconstrained map between language representations, and neither asks whether the model already holds the answer and simply fails to say it. It largely does: a linear probe recovers the truth from the very activations the model fails to express. We propose RoSh, a per-language shift and rotation of the residual stream, computed in closed form at three layers, with no training and no weight modified. It improves every model and closes 75% of the gap on average, helping most where the model was worst: Arabic on Llama-3B goes from chance to nearly the English level, and a fifth fewer of the claims answered correctly in English are lost in translation. What remains is no longer a read-out failure: afterwards the head recovers as much of what is encoded outside English as it does in English. An unconstrained map fitted on the same pairs falls below the untouched baseline, so the orthogonality constraint is doing the work, and every model clears a scrambled-correspondence control and ten further controls. On the two benchmarks of the closest inference-time method, latent-space intervention, run with its own data and metric code, RoSh's gains are five to thirteen times larger.

View source

Similar papers

#natural language process... Open access Sep 2026

Does CoT-Pass@k Really Check the CoT? A Multilingual Mathematical Audit

Pass@k measures whether a model reaches a correct answer under repeated sampling, but never how: a lucky guess counts the same as sound reasoning. CoT-Pass@k was proposed to close that gap, adding an LLM-as-judge that must assess a solution's reasoning chain before it counts. Its value rests entirely on one assumption:...

Tarık Tuna Taşaltı, Burcu Hudaverdi, David Semedo · 0 citations
#natural language process... Preprint Sep 2026

Do LLMs Make More Mistakes If They Do Not Believe the Input Data?

How faithfulness of LLMs to provided context depends on how plausible they perceive the context to be (context-memory conflict) is analyzed, which makes use of the increased difficulty of non-English and low-resource language text generation and input data based on local knowledge.

Peter Kochelka, Aleš Manuel Papáček, Vojtěch Dvořák et al. · 0 citations
Preprint Aug 2026

Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs

J-PragEval-v0 is introduced, a minimal-pair benchmark isolating four such phenomena from surface fluency, and Pragmatic Representation Steering is specified, a parameter-free inference-time method that edits residual-stream activations along the class-mean-difference directions probing identifies.

F. Braun · 1 citation
#machine learning Preprint Sep 2026

Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models

When a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data. Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not. This paper removes the guessing. Two model...

Arman Nik Khah · 0 citations
#artificial intelligence Preprint Sep 2026

Does a model's stated reason for rejecting a candidate do any work?

Asked to choose between candidates and explain the choice, a language model often rejects a rival by naming a fact its profile lacks: no director, no date of death. That sentence is a claim about the text in front of the model, and it can be tested without any judge. We insert a real corpus sentence stating the named f...

Archit Rastogi · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.