Surprisal from Large Language Models as an individual difference: Capturing the individual regardless of the text answer

From Exposure to Expectation: Frequency, Surprisal, and Language Across Development in Spanish

Surprisal, the negative log-probability a language model assigns to a word given its preceding context, reliably predicts adult reading times. Does it contribute as much to explaining when children acquire individual words? Frequency reflects a learner's cumulative exposure to a word, whereas surprisal reflects how pre...

Large Language Models as Distributional Baselines for Language Tasks

The central contributions of this paper articulate the conditions under which distributional predictability threatens the internal validity of an experiment and provide concrete recommendations for how to control for this potential confound.

Proceedings of the Annual Meeting of the Cognitive Science Society

Chronologic: Measuring Language Models'Ability to Represent the Past

Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its responses fit the period represented. Validation is challenging, because this is not a task living people ordinarily perform, and because many questions have multiple correct an...

Modeling the Memory-Surprisal Trade-Off over Time: Communicative Efficiency Decreases with Lexico-Grammatical Change in Scientific English

Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics

Contextual and unigram surprisal are standard predictors of reading time in computational psycholinguistics. While contextual surprisal is typically derived from neural language models, unigram surprisal is usually obtained from independent corpus counts. Although both quantities are deterministic functionals of a sing...