Skip to content
Preprint

Hidden Language Consistency Phenomena in Reasoning LLMs

Aug 2026 · 0 citations · 39 references
Computer Science

TL;DR

The results show that multilingual capability cannot be characterized by accuracy alone; reliable evaluation should jointly consider task accuracy, language consistency, and task difficulty for multilingual benchmarks.

Abstract

Multilingual reasoning models are commonly evaluated by whether they arrive at the correct answer, but not by whether they preserve the intended language while reasoning and responding. This omission conceals important multilingual behaviors that emerge as tasks become harder. In this paper, we study task difficulty, task accuracy, thinking-language consistency (TC), and answer-language consistency (AC) across reasoning models using PolyMath benchmark in eight languages and four difficulty levels. We uncover four findings: (1) language consistency exhibits four difficulty-dependent behaviors: output-language consistency remains aligned with input, remains misaligned, degrades gradually, or collapses abruptly. (2) We identify the language consistency breakdown effect, where increasing difficulty can cause a sudden drop in output-language consistency, especially in less strongly represented and non-Latin-script languages. (3) Due to this breakdown effect, accuracy can be preserved or even improved at a harder difficulty level as the model shifts to its internal dominant language. (4) Quantization can improve or degrade output-language consistency independently of its effect on accuracy, with GPTQ and AWQ often outperforming AutoRound under tolerance-based voting with {\epsilon} = 1.0. These results show that multilingual capability cannot be characterized by accuracy alone; reliable evaluation should jointly consider task accuracy, language consistency, and task difficulty for multilingual benchmarks.

View source

Similar papers

Preprint Aug 2026

Skill Issue: Are Skills Language-Invariant in LLMs?

This work quantifies cross-lingual skill inconsistency orthogonally from knowledge and general benchmark performance via multilingual self-play, and shows that skill discrepancies are a measurable major roadblock in the development of truly multilingual models.

Bobby Cheng, Adam Gaber, Zhengzhe Liu et al. · 1 citation

What Does Post-Training Change in Multilingual Reasoning?

These stages establish a constructive post-training path from English-pivoted capability to multilingual reasoning that is reliably delivered, and jointly track correctness, language adherence, termination, and delivery efficiency.

Hong-Yang Li, Xiao Li, Caesar Wu et al. · 0 citations
#natural language process... Preprint Sep 2026

Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning

It is shown the path to generalizing L2 reasoning to held-out languages goes through broader language coverage, readily available multilingual non-reasoning data, and a sufficient English reasoning backbone, indicating that reasoning is a language-agnostic behavior that can be transferred across typologically diverse l...

Mehrnaz Mofakhami, Ananya Sahu, Alejandro R. Salamanca et al. · 0 citations
#machine learning Preprint Oct 2026

A Systematic Study of Small Language Models on Abstract Reasoning Tasks

Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-specific regularities. We study this distinction in small language models on the ARC-TGI benchmark, which organizes abstract grid transformations into controllable task famili...

Nur A. Zarin Nishat, Jens Lehmann, Andrei C. Aioanei et al. · 0 citations
#natural language process... Open access Sep 2026

Choosing the Right Language Mode at Inference Time for Multilingual Reliability

Reliability-Aware Adaptive Inference (RAAI), a training-free test-time framework that performs Expected Calibration Error-aware routing and prompt fusion, and uses a mid-layer Risk Index to gate sequential reasoning, allocating compute only when it is likely to help and suppressing harmful bilingual redundancy, is addr...

Ekata Mitra, Ameeta Agrawal · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.