Skip to content

Aligned Data Can Induce Misalignment via Context Confusion

Sep 2026 · 1 citation · 22 references
Computer Science

TL;DR

It is argued that it is difficult to predict the alignment state of a model after training by inspecting the training data alone, which highlights the importance of comprehensive post-training alignment evaluations.

Abstract

Large language models (LLMs) are frequently updated for various use cases, where filtering out misaligned training samples is a common practice for preventing post-update misalignment. However, alignment is inherently context-dependent: a recommendation that is aligned in one context may be inappropriate in another. For example, in response to the question"What should a researcher do with the research data?", recommending that the researcher preserve the data for reproducibility is aligned. In contrast, recommending data saving in response to"What should a mobile-app developer do with users'sensitive data?"may be inappropriate from a privacy perspective. Starting from this observation, we identify a post-training phenomenon where aligned training induces misaligned behavior in other contexts. We call this phenomenon **context confusion**. We demonstrate context confusion across three domains: (1) Gender Equality, (2) Privacy, and (3) Physical Safety. We further show that context confusion causes narrow misalignment, in contrast to emergent misalignment, and is not effectively reduced by injecting general alignment data, but can be substantially reduced by including targeted alignment data for the misaligned domain or providing in-context learning examples during inference. Lastly, we provide a mechanistic explanation of *context confusion*. We observe that queries from different domains can undergo similar representational shifts during the fine-tuning. Consequently, a query from a different domain may activate the same behavioral feature learned during fine-tuning, which causes the behavior to transfer to a context where it is misaligned. Based on our findings, we argue that it is difficult to predict the alignment state of a model after training by inspecting the training data alone, which highlights the importance of comprehensive post-training alignment evaluations.

View source

Similar papers

#artificial intelligence Preprint Aug 2026

Emergent Misalignment Is Not Magical

The EM generalization metric is extended from a scalar distance to a dataset-specific generalization direction, which robustly predicts EM models'evilness under semantics-preserving prompt perturbations including appending random tokens and paraphrasing, where other methods do not reliably generalize.

Ming-Xuan Li, Qirun Dai, Hesi Wang et al. · 0 citations
#artificial intelligence Preprint Sep 2026

TAME: Token Attribution and Masking for Emergent misalignment

Fine-tuning an aligned language model on narrow, flawed data can induce harmful behavior far outside the training domain, known as emergent misalignment (EM). Prior work has localized EM in model weights, activations, and training documents, but it remains unclear which training tokens carry the relevant fine-tuning si...

Md Rayhanul Masud, Md. Rizwan Parvez · 1 citation
#artificial intelligence Preprint Sep 2026

Alignment Forecasting: Predicting Misalignment From Training Data

Training a language model on data with a narrow flaw can sometimes make the model broadly misaligned. Inspecting the data at face value often does not settle whether it will emerge, and today it is caught only after training, by auditing the resulting model. To complement post-hoc audits, we introduce Alignment Forecas...

Chen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky et al. · 0 citations
#artificial intelligence Preprint Sep 2026

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment

This work uses training data attribution to quantitatively estimate how much each harmful example contributes to EM and benchmark the quality of the attribution via retraining -- a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score.

Gonçalo Paulo, Louis Jaburi, Nora Belrose et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.