Sep 2026· Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing· Vol abs/2609.32119· 0 citations· 82 references
Computer Science
TL;DR
This work systematically vary the context window size of GPT-2 across four large-scale naturalistic English reading-time datasets and observes a U-shaped relationship, demonstrating that tracking long-range coreference relations is one important factor for the alignment between LM surprisal and human reading behavior.
Abstract
Language models (LMs) are often used as a tool to model human language processing. Recent studies suggest that severely restricting LMs'context window improves their fit to human psycholinguistic data by simulating human working memory constraints. However, it is possible that this strict memory-decay approach overlooks humans'reliance on long-range structural representations, such as discourse structre. In this work, we systematically vary the context window size of GPT-2 across four large-scale naturalistic English reading-time datasets and observe a U-shaped relationship: Although restricted contexts (<20 tokens) successfully capture local memory limitations, expanded contexts (500--1,000 tokens) ultimately yield the highest overall psycholinguistic fit. To investigate the mechanism driving this benefit, we conduct a counterfactual inference-time experiment that disrupts cross-sentential entity chains by pronominalizing repeated discourse entities. Obscuring these structural linkages significantly degrades the predictive power of larger context windows by 20% to 40%. Our experiments demonstrate that tracking long-range coreference relations is one important factor for the alignment between LM surprisal and human reading behavior, and approximate the extent to which human comprehenders use global discourse relations during language processing.
We reproduce and stress-test the work of Yu et al. (2023), who characterize how language models (LMs) arbitrate between memorized knowledge and contradictory in-context statements. We replicate their world-capitals experiments on 31 models spanning Pythia, GPT-2, Qwen3, and Ministral families, including base and post-t...
Guilhem Fouilhé, Nicholas Asher, Philippe Muller· Proceedings of the 9th Black...· 0 citations
Abstract Memory limitations have been assumed to be a major factor shaping human sentence comprehension, but characterizing memory-demanding structures in a broad-coverage and cross-linguistic manner has remained a challenge. A recent study suggested that such a general characterization can be obtained by assuming effi...
Shinnosuke Isono, Kohei Kajikawa, Yohei Oseki et al.· Open Mind· 2 citations
Systematicity, the probabilistic mapping of form to meaning, permeates language at all levels, and sublexical cues have been shown to govern human pseudoword processing. Yet whether LLMs exhibit comparable sensitivity to these cues remains unclear. We tested five LLMs on two Italian two-alternative forced-choice pseudo...
Jing Chen, Giulia Loca, Simona Amenta et al.· 0 citations
We introduce STRUCTURALCOST, a self-paced reading dataset of 475 participants and 40,800 observations isolating the processing cost of long-distance subject-verb dependency resolution. We replicate a low-powered psycholinguistic finding at NLP scale, namely that human reading times at the main verb increase with depend...
Nina Nusbaumer, Iria de-Dios-Flores, Corentin Bel et al.· Conference on Empirical Meth...· 0 citations
Some properties of languages, e.g., subject-object-verb (SOV) word order, are more prevalent than others among the thousands of attested natural languages (NLs). Such typological commonality is often attributed to learning biases. Computational simulations, recently with language models (LMs), have facilitated the expl...
A systematic comparison of multiple attention-based memory mechanisms across different model sizes and training corpora finds that constraints that are sensitive to the content of intervening tokens consistently achieve the highest alignment with human reading times, outperforming distance-based constraints.
Lanni Bu, Xiulin Yang, Christian Clark et al.· 0 citations
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.