Language as a Hidden Variable: Measuring Behavioral Divergence in Multilingual Large Language Models

Abstract

Multilingual large language models (LLMs) are widely assumed to behave consistently regardless of the language in which a prompt is expressed. We challenge this assumption through a controlled empirical study in which language itself is isolated as the sole independent variable. We present 30 semantically equivalent prompts across three prompt categories—Factual, Normative, and Safety-sensitive—to two state-of-the-art LLMs (Llama-3.3-70B and GPT-OSS-20B) in English, Hindi, and French, yielding 360 responses. We introduce the Behavioral Divergence Score (BDS), a composite metric combining semantic similarity (SBERT cosine distance), sentiment divergence, and refusal-class mismatch. Our analysis reveals that cross-language divergence consistently exceeds the intra-language noise floor by factors of 1.1×2.5×, that normative prompts produce the highest sentiment divergence $\left(\bar{S}_{\text{sent}}=0.28\right)$, and that safety-category refusal behaviour is inconsistent across languages in 10% of prompt-model pairs. All three pre-registered directional hypotheses were not supported in strict statistical sense, a result we interpret as a meaningful empirical finding: the divergence is real and category-dependent, but it does not follow monotonic ordering predicted by the linguistic-distance theory alone. These results motivate rethinking multilingual evaluation beyond aggregate accuracy metrics and highlight concrete safety implications for real-world deployments.