Zhong-Jiang He

Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models

Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed. This interference is especially consequential in long, packed, or distractor-heavy contexts, where useful evidence and irrelevant spans co...