Future-Proof Data Systems
TL;DR
A line of work that treats future-proofness as an abstraction-design problem is summarized, identifying boundaries across the data processing stack where concerns must evolve independently, and design interfaces that preserve enough structure across that boundary for optimization.
Abstract
Database systems are increasingly expected to support evolution in data workloads, representations, and hardware environments. Existing designs often respond through either specialization, which can deliver high performance but leads to narrow and brittle systems, or extensibility, which broadens functionality but can produce opaque extensions and growing system complexity. This paper summarizes a line of work that treats future-proofness as an abstraction-design problem: we identify boundaries across the data processing stack where concerns must evolve independently, and design interfaces that preserve enough structure across that boundary for optimization. We illustrate this at three levels of the stack. At the level of data structures, Sortledton unifies analytics, pattern matching, and transactional updates on a single graph representation. At the level of data formats, AnyBlox allows engines to read datasets that provide sandboxed self-decoding logic, reducing the need for bespoke readers. At the level of query processing, LingoDB and its declarative sub-operator layer recast query optimization as a sequence of extensible compiler passes, enabling cross-domain optimization and hardware-specific lowerings.