Copy2: Efficient Data Transfer in High Performance Environments

Abstract

Efficient data movement is a critical bottleneck in High Performance Computing (HPC) centers, especially for large-scale migrations across parallel and distributed storage systems. Existing tools such as rsync and rclone often underutilize network and metadata bandwidth, provide limited hard link preservation, and offer no guarantees on memory usage, limiting their effectiveness for HPC environments. Motivated by a large-scale migration of data at Partnership for an Advanced Computing Environment (PACE), we present copy2, a high-performance, resource-aware data transfer tool for modern HPC storage such as Lustre and VAST Data. copy2 features a scalable metadata crawler, inode-based hard link preservation, asynchronous high-throughput transfers, and bounded memory allocation. Preliminary results show up to twice the throughput of rclone with substantially lower energy usage, and successful migration of projects with hundreds of millions of files that would be impractical with existing tools. copy2 offers HPC administrators and researchers a robust, predictable, and efficient solution for large-scale data management.