Results show that selective anti-interference is an effective objective-level signal for robust context use and identifies locally sufficient tokens and sharply reduces prefix sensitivity on gate-selected suffix tokens.
Abstract
Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed. This interference is especially consequential in long, packed, or distractor-heavy contexts, where useful evidence and irrelevant spans coexist. We propose Selective Prefix Anti-Interference Regularization (SPAR), a pretraining objective for selective anti-interference. SPAR runs the original sequence and a corrupt-prefix input in which only the far prefix is changed, then uses a short-context sufficiency gate and a gated KL objective to stabilize locally supported suffix predictions. The gate operationalizes a model-based estimate of whether the far prefix supplies additional information about the target token. Mechanism analyses show that the gate identifies locally sufficient tokens and sharply reduces prefix sensitivity on gate-selected suffix tokens. In continued training on pretrained base models, SPAR improves RULER across Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-1B, Llama-3.1-8B, and GPT2-XL under equal counted training compute; pretraining experiments further show gains on both RULER and NoLiMa. These results show that selective anti-interference is an effective objective-level signal for robust context use.
Under a softmax retrieval abstraction, a finite-sample upper bound is derived showing that maintaining a fixed accuracy target above base rate requires the evidence margin to scale as $\Omega(\sqrt{\log N})$, where N denotes the effective distractor count rather than necessarily the raw context length.
Meysam Ghaffari, Nina Fatehi, Bhaskar Sen et al.· 0 citations
The results show that early abstract training can reliably shape the capabilities language models later acquire, and in small language models, such a warm-up improves specific capabilities that are not reflected in language-modeling perplexity.
HearInContext is introduced, a Mandarin-English benchmark that pairs shared synthetic speech with assistant replies supporting different interpretations that extends to explicit conditions excluded from fine-tuning and to Mandarin hotword recognition on real recordings.
A Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces, which is substantially more token-efficient than forgetting-aw...
Zheng-Hao Zhao, Gao-Wen Liu, Zhi-Ling Lan et al.· 0 citations
Answer-Convergence Stopping (ACS) is introduced, a training-free stopping rule that measures rather than asks, and reveals that by properly utilizing the output signals of frozen models, it can achieve favorable behaviors like adaptive stopping without the need for additional training.
It is found that context does not simply erase memorization, and many samples that are extractable without context remain extractable under the retrieved context, especially as the prefix length increases.
Ali Satvaty, Narjes Sharafi, Jirui Qi et al.· Proceedings of the 2026 Conf...· 0 citations
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.