Where Should Compaction Run? Compaction Worker Selection on Disaggregated LSM-KVS

Abstract

In disaggregated storage architectures, compaction in LSM-trees is the dominant source of network traffic. Because every input SST must be fetched over the network, write amplification multiplies this data volume, leading to a measured network amplification (NA) of 8.72 × in our evaluation. Although existing remote-compaction systems decouple compaction from the compute node, they select workers without considering physical data placement, failing to mitigate this network bottleneck. We ask a question that prior work has not addressed: does physical data placement create a locality opportunity at the per-compaction level, and can it be exploited for worker selection? Through measurement on an 8-node, 32-OSD Ceph cluster, we show that while data placement is globally uniform, the input data for any single compaction is typically skewed across nodes—so for almost every compaction task, a specific node holds a disproportionate share of its input. Motivated by this finding, we prototype LARC (Locality-Aware Remote Compaction), which evaluates candidate workers on the client side using a cached CRUSH map and routes each task to the highest-scoring worker. Preliminary results on RocksDB v9.10.0 show that LARC raises the average per-compaction locality to \(51.1\%\), well above the \(R/N = 37.5\%\) analytical expectation, increases fillrandom throughput by \(16.1\%\), and reduces NA by \(32.8\%\) compared to the Local baseline.