Scalable Entity-Centric Data Filtration for LLM Pretraining

Abstract

Large language models rely on scaling both model parameters and data, but increasing dataset size significantly raises computational and storage costs. While existing data filtering approaches remove low-quality or duplicated content, further refinement is needed to improve pretraining efficiency. This work introduces an entity-centric filtering framework that leverages entity-based representations to identify and remove redundant or low-density information. By using entities as compact semantic anchors, the method provides an interpretable signal of semantic content across documents. Experiments on English, Russian, and Chinese demonstrate that pruning 25% of an already filtered, high-quality corpus by removing rare documents identified through entity-based clustering consistently improves model performance. Rarity is defined as corpus-level underrepresentation rather than low quality: the removed documents belong to sparsely populated entity-based clusters and remain comparable to the retained data in repetition, readability, and perplexity. Under a matched 75% retention budget, the proposed approach outperforms language-model-based document scoring, SemHash, SemDeDup, and random selection, as well as training on the full corpus. The deterministic, dictionary-based extraction procedure scales near-linearly and incurs negligible filtration cost relative to the resulting reductions in pretraining compute and storage.