Jul 2026· International Symposium on Communication Systems, Networks and Digital Signal Processing· pp. 1-6· 0 citations· 15 references
TL;DR
A queueing network (QN) model of the rebalance process of the process used in MongoDB is developed, and a preliminary evaluation of performance prediction by the QN model via simulations is described, to understand the impact of different parameters of the rebalance process.
Abstract
Scalable storage systems typically provide mechanisms for re-distributing data over time to ensure a balanced storage allocation. Such mechanisms have traditionally been considered as background activities meant to run at low priority, to avoid penalizing applications accessing storage. In this work, we focus on an alternative design point where data re-distribution is run as a high-priority activity, meant to re-balance data at the full speed allowed by the newly added resources. Such rapid redistribution has become viable in recent years due to the rise of low-overhead technologies in the networking and disk storage space. A challenge in such a scenario is to schedule data transfer flows to newly added nodes in a way that fully and efficiently utilizes network, CPU, and disk resources in the new nodes, while avoiding overload. To understand the impact of different parameters of the rebalance process, such as the number of simultaneous senders, we develop a queueing network (QN) model of the process used in MongoDB, and describe a preliminary evaluation of performance prediction by the QN model via simulations.
Results show up to 2.3x lower remote-access overhead compared to TCP-based designs, a 13% increase in cluster-wide memory utilization at rack scale, and up to 63% reduction in miss rate for memory-constrained tenants under skewed workloads.
Data processing clusters dealing with big data are facing extended completion times for jobs because the RDMA feature is not being used efficiently. Our production data from a large cluster, which has many server nodes and is responsible for processing extensive data jobs, reveals that the current use of the RDMA technology is causing some jobs to finish much later than usual, with a few taking more than double the standard time to be completed. In this paper, we introduce the design and implementation of Turbo, a high-performance, scalable communication framework tailored for large-scale data processing clusters. The essence of Turbo’s strategy lies in the use of a dynamic block-level flowlet transmission system and a non-blocking communication middleware, which are designed to boost network throughput and system scalability. Moreover, Turbo maintains high system reliability by incorporating an external shuffle service with TCP as a fail-safe option and maintaining metadata management information using the NICs. We have integrated Turbo into Apache Spark and conducted evaluations on both a small-scale test environment and a large-scale cluster with hundreds of server nodes. The findings from the small-scale testbed demonstrate that Turbo enhances network throughput by 15.1% and upholds high system reliability. Additionally, the large-scale production data indicates that Turbo is capable of decreasing job completion times by 23.9% and increasing the job completion rate by $2.03\times $ compared to the current RDMA solutions. In addition, during large-scale tests, we also found that Turbo has improved the computing efficiency of the cluster and saved approximately 24.3% of the CPU utilization.
Xuya Jia, Zhiyi Yao, Chao Peng et al.· IEEE Transactions on Network...· 0 citations
Efficient data movement is a critical bottleneck in High Performance Computing (HPC) centers, especially for large-scale migrations across parallel and distributed storage systems. Existing tools such as rsync and rclone often underutilize network and metadata bandwidth, provide limited hard link preservation, and offer no guarantees on memory usage, limiting their effectiveness for HPC environments. Motivated by a large-scale migration of data at Partnership for an Advanced Computing Environment (PACE), we present copy2, a high-performance, resource-aware data transfer tool for modern HPC storage such as Lustre and VAST Data. copy2 features a scalable metadata crawler, inode-based hard link preservation, asynchronous high-throughput transfers, and bounded memory allocation. Preliminary results show up to twice the throughput of rclone with substantially lower energy usage, and successful migration of projects with hundreds of millions of files that would be impractical with existing tools. copy2 offers HPC administrators and researchers a robust, predictable, and efficient solution for large-scale data management.
Aiden Lambert, Deepa Panish· Practice and Experience in A...· 0 citations
Parallel file systems in HPC manage metadata for billions of files across distributed storage servers. The structure of this metadata, how files distribute by size, how users concentrate across servers, which storage policies are actually in use, determines operational decisions about capacity planning, load balancing, and data migration. Despite decades of HPC storage research, these structural properties remain underreported in the published literature for HPC parallel file systems. Runtime I/O behavior has been profiled extensively at the application level. Aggregate monitoring captures quotas and throughput. But the metadata that accumulates on disk, the artifact of all user activity over the life of a system, has received little systematic study. Existing tools can extract inode-level detail in principle, but each imposes barriers that discourage routine analysis: online queries load the metadata server, database replicas require ETL pipelines, and low-level utilities demand scripting that few administrators undertake. Most sites already take periodic MDT backup images for disaster recovery, yet these images sit unused between incidents. We propose using them for periodic metadata analysis. The method parses Lustre-specific on-disk structures and exposes them as SQL-queryable tables, requiring no intermediate files and placing no load on the metadata server. We implemented this method as a DuckDB extension and applied it to a Lustre deployment during its early access period. The analysis surfaces operationally relevant patterns: inode counts differ 4.9:1 across MDTs due to user behavior, 99.9997% of files use the default stripe configuration, the Python package manager uv created 69,785 cross-directory hard links on one MDT, and an automated process generated a 129,000-entry directory in under two minutes. We present these findings alongside the monitoring items they motivate, providing a reusable template for periodic metadata analysis. The tool is available at https://github.com/tsukuba-hpcs/lustre-query.
Sohei Koyama, Osamu Tatebe· IEEE International Symposium...· 0 citations
With the emergence of data-intensive computing, which is due to the growth of the data produced and generated each day, it became necessary to store and manage big data. Cloud data storage is actually the best choice for large distributed systems. Successful Cloud Computing cannot be achieved without a reliable data-management system to store and handle the enormous volume of data. Management of the available storage system at large scale becomes progressively more complicated, and we face many challenges, such as scalability, data availability, fault tolerance, etc. Also, data storage is faced with specific access patterns: highly concurrent reads of data from the same file, many overwrites, and very concurrent appends to the same file. Most of the existing storage systems use versioning to bring and enhance data access parallelism and this enables better performance levels under concurrency; but, generally, these systems use one component (version manager), which is responsible for generating new versions of each file stored. When we speak in the context of big data, the requests for read, write and append increase. If these requests are managed by a single component, then we have a performance bottleneck and an overloaded version manager. To avoid this drawback, we proposed and designed a new architecture of storage systems that uses versioning; the new architecture uses multi-version managers to support better the scalability and provide partial fault tolerance. To illustrate the practicability of our approach, we assessed it on the BlobSeer data-management system. The experimental results demonstrate that our architecture achieves near-linear scalability for CREATE operations (495 ops/s per additional version manager), reduces WRITE execution time by up to 66%, and maintains 67% availability under single-node failures, all while introducing minimal resource overhead (3% aggregate CPU increase). These results confirm that the proposed multi-version manager architecture offers a practical, scalable, and partially fault-tolerant solution for Cloud data-storage systems.
Baya Chalabi, Y. Slimani· Future Internet· 0 citations