A large-scale pipeline for LLM-assisted corpus annotation

Abstract

As natural language corpora expand at an unprecedented rate, manual annotation remains a significant methodological bottleneck in corpus linguistic work. We address this challenge by presenting a scalable pipeline for automating grammatical annotation using large language models (LLMs). We demonstrate the pipeline’s accessibility and effectiveness through a diachronic case study of variation in the English evaluative consider construction ( consider X as / to be /Ø Y). We annotate 143,933 ‘consider’ concordance lines from the Corpus of Historical American English (COHA) via the OpenAI API in under 60 hours, achieving 98%+ accuracy on two sophisticated annotation procedures. Subsequent analysis reveals previously undocumented genre-specific trajectories of change, enabling us to advance new hypotheses about the relationship between register formality and competing pressures of morphosyntactic reduction and enhancement. Our results suggest that LLMs can perform a range of data preparation tasks at scale with minimal human intervention, unlocking substantive research questions previously beyond practical reach.