Skip to content

Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models

Aug 2026 · 0 citations · 34 references
Computer Science

TL;DR

Results show that selective anti-interference is an effective objective-level signal for robust context use and identifies locally sufficient tokens and sharply reduces prefix sensitivity on gate-selected suffix tokens.

Abstract

Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed. This interference is especially consequential in long, packed, or distractor-heavy contexts, where useful evidence and irrelevant spans coexist. We propose Selective Prefix Anti-Interference Regularization (SPAR), a pretraining objective for selective anti-interference. SPAR runs the original sequence and a corrupt-prefix input in which only the far prefix is changed, then uses a short-context sufficiency gate and a gated KL objective to stabilize locally supported suffix predictions. The gate operationalizes a model-based estimate of whether the far prefix supplies additional information about the target token. Mechanism analyses show that the gate identifies locally sufficient tokens and sharply reduces prefix sensitivity on gate-selected suffix tokens. In continued training on pretrained base models, SPAR improves RULER across Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-1B, Llama-3.1-8B, and GPT2-XL under equal counted training compute; pretraining experiments further show gains on both RULER and NoLiMa. These results show that selective anti-interference is an effective objective-level signal for robust context use.

View source

Similar papers

#artificial intelligence Preprint Aug 2026

Context Poisoning as Extreme-Value Attention Interference in Long-Context Language Models

Under a softmax retrieval abstraction, a finite-sample upper bound is derived showing that maintaining a fixed accuracy target above base rate requires the evidence margin to scale as $\Omega(\sqrt{\log N})$, where N denotes the effective distractor count rather than necessarily the raw context length.

Meysam Ghaffari, Nina Fatehi, Bhaskar Sen et al. · 0 citations

Lasting Effects of Abstract Pretraining Beyond Perplexity

The results show that early abstract training can reliably shape the capabilities language models later acquire, and in small language models, such a warm-up improves specific capabilities that are not reflected in language-modeling perplexity.

Zachary Shinnick, Hemanth Saratchandran, Damien Teney et al. · 1 citation
#natural language process... Preprint Sep 2026

HearInContext: A Benchmark for Implicit Context in Speech Recognition

HearInContext is introduced, a Mandarin-English benchmark that pairs shared synthetic speech with assistant replies supporting different interpretations that extends to explicit conditions excluded from fine-tuning and to Mandarin hotword recognition on real recordings.

Yi-Fan Gao, Yao Tian, Hong-Bin Suo · 0 citations
#machine learning Preprint Oct 2026

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

A Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces, which is substantially more token-efficient than forgetting-aw...

Zheng-Hao Zhao, Gao-Wen Liu, Zhi-Ling Lan et al. · 0 citations

The Model Knows When to Stop: Training-Free Early Stopping for Long-Context Reading

Answer-Convergence Stopping (ACS) is introduced, a training-free stopping rule that measures rather than asks, and reveals that by properly utilizing the output signals of frozen models, it can achieve favorable behaviors like adaptive stopping without the need for additional training.

Muath Alyobi, Mohamed Eltahir, Almoayyad Abuljdail et al. · 0 citations
#artificial intelligence Conference Open access Oct 2026

Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes

It is found that context does not simply erase memorization, and many samples that are extractable without context remain extractable under the retrieved context, especially as the prefix length increases.

Ali Satvaty, Narjes Sharafi, Jirui Qi et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.