DdlRT: A deterministic data layout for efficient redundancy transitioning in erasure-coded systems

Abstract

Erasure coding is widely used in distributed storage systems because it provides high reliability with low storage overhead. To adapt to changing performance and reliability requirements, systems may perform redundancy transition on erasure-coded data, i.e., re-encoding existing data on the fly with new coding parameters. However, existing random data layouts incur substantial transition traffic and severe load imbalance. We present DdlRT, a deterministic data layout for redundancy transition in distributed storage systems. DdlRT leverages two orthogonal arrays to build a two-level mapping across racks and nodes, deterministically placing stripes on designated storage nodes. By making data layout transition-aware, DdlRT avoids or significantly reduces transition traffic while preserving load balance at both the rack and node levels. Large-scale simulations and real-system experiments show that, compared with state-of-the-art schemes, DdlRT reduces redundancy transition traffic by 68.13%–92.37% and transition time by 38.32%–94.35%, while also improving load balance.