Beyond Context Windows: Data Transformations at Scale with LLMs
TL;DR
A framework for scalable, dataset-scale LLM-based data transformation that processes data row-by-row in batches over lazily evaluated, distributed dataframes, enabling LLM-based transformations on arbitrarily large datasets without loading them into memory.
Abstract
: Applying large language models (LLMs) to real-world tabular datasets at scale is subject to a fundamental mismatch between model context limits and enterprise data size. Existing approaches address this by truncating data or fitting everything into a single prompt — methods that do not scale and sacrifice row-level semantic understanding. We present a framework for scalable, dataset-scale LLM-based data transformation. It processes data row-by-row in batches over lazily evaluated, distributed dataframes, enabling LLM-based transformations on arbitrarily large datasets without loading them into memory. Three composable primitives cover the most common patterns: MAP for field-level enrichment, FILTER for row selection, and REDUCE for semantic deduplication via embedding-based similarity search. An agentic orchestration layer accepts a plain-language goal, plans the required primitive sequence, and routes operations that require no semantic reasoning to generated Python code. We evaluate the framework on five domain-specific benchmarks. On a 51,291-row agentic enrichment task, the pipeline achieves 96.2% accuracy in 9.4 minutes. On a 3-million-row deduplication task, REDUCE completes in 1.90 hours. Single-primitive benchmarks reach 99.6% coverage on category extraction, Macro F1 of 0.97 on sentiment classification, and 96% accuracy on compliance filtering. The framework is released as open source at https://github.com/vitalops/datatune.