AutoLiquid: Autonomic Data Layout Optimization for the Databricks Lakehouse

Abstract

Data layout recommendation is a classic problem in database literature, yet the downstream challenge of autonomously applying these changes in live production systems is less frequently studied. In Lakehouse architectures, selecting the right clustering key is critical for data skipping performance, but remains a manual process that cannot scale to the hundreds of millions of tables managed by platforms like Databricks. We present AutoLiquid, an autonomic system that solves this by continuously selecting, verifying, and applying clustering keys for Liquid Clustering tables, requiring only a single "CLUSTER BY AUTO" declaration from the user. To navigate the tension between adapting to unpredictable workloads and maintaining strong performance guarantees against regressions, AutoLiquid leverages the decoupled storage and compute architecture of the Lakehouse. It utilizes lightweight heuristics over collected scan telemetry to identify promising candidate keys, followed by a fast shadow verification process that measures actual pruning benefits on sampled data. This "verify before commit" approach ensures that only decisions with demonstrated improvement are applied. We present the algorithm and end-to-end verification flow, providing empirical results using real customer workload data. When compared to customer-selected clustering keys, AutoLiquid achieves similar or better performance on over 95% of evaluated tables. AutoLiquid is currently deployed in production at Databricks, managing clustering keys for millions of tables.