The I/O Resource Manager (IORM), a multi-stage distributed scheduler deployed in production within Oracle Exadata Exascale, combines three mechanisms: a hardware-aware cost modeler that normalizes I/O accounting using datasheet-derived fixed costs to make limits invariant across hardware generations.
Abstract
Shared-nothing disaggregated storage clusters that serve both latency-sensitive databases and opaque block-volume workloads face two governance problems unsolved by existing schedulers: maintaining consistent performance across heterogeneous hardware generations, and enforcing global I/O limits when access patterns skew to a subset of storage nodes. We present the I/O Resource Manager (IORM), a multi-stage distributed scheduler deployed in production within Oracle Exadata Exascale. IORM combines three mechanisms: a hardware-aware cost modeler that normalizes I/O accounting using datasheet-derived fixed costs to make limits invariant across hardware generations; a quantum-based rate limiter with bounded carry-forward credits that accommodates database micro-bursts while enforcing long-term SLOs; and a distributed adaptive feedback controller that redistributes unused entitlements across the cluster to resolve topological access skew. Beyond design, we share operational lessons from production deployment. On an 8-node test cluster running up to 100 concurrent tenant volumes, IORM converges within 5\% of provisioned limits under extreme sequential skew, scales without inter-tenant interference, and recovers full throughput within 15 seconds of a storage-node failure.
Elastic block storage (EBS) with the storage-compute disaggregated architecture is a key component in modern cloud infrastructure. EBS offers users storage resources in the form of elastic solid-state drives (ESSDs). Nonetheless, despite recent efforts that have documented EBS architectures from the provider's perspective, how ESSDs perform differently from local SSDs and how host software should adapt accordingly have not been sufficiently studied. In this paper, we conduct a user-centric, black-box performance characterization of ESSDs from Amazon AWS and Alibaba Cloud. We make three main contributions: (1) an ESSD contract that presents four behavioral observations and five actionable implications for software adaptation, (2) a refined I/O rate-limiting model combining bandwidth-IOPS dual limiting and fine-grained token refilling to suppress latency spikes, and (3) a case study on RocksDB that derives four guidelines on cache management, I/O regulation, storage budget utilization, and compression algorithms. Collectively, we hope these contributions can serve as a practical reference for EBS users to understand and exploit the distinctive performance properties of ESSDs.
Results show up to 2.3x lower remote-access overhead compared to TCP-based designs, a 13% increase in cluster-wide memory utilization at rack scale, and up to 63% reduction in miss rate for memory-constrained tenants under skewed workloads.
Parallel file systems in HPC manage metadata for billions of files across distributed storage servers. The structure of this metadata, how files distribute by size, how users concentrate across servers, which storage policies are actually in use, determines operational decisions about capacity planning, load balancing, and data migration. Despite decades of HPC storage research, these structural properties remain underreported in the published literature for HPC parallel file systems. Runtime I/O behavior has been profiled extensively at the application level. Aggregate monitoring captures quotas and throughput. But the metadata that accumulates on disk, the artifact of all user activity over the life of a system, has received little systematic study. Existing tools can extract inode-level detail in principle, but each imposes barriers that discourage routine analysis: online queries load the metadata server, database replicas require ETL pipelines, and low-level utilities demand scripting that few administrators undertake. Most sites already take periodic MDT backup images for disaster recovery, yet these images sit unused between incidents. We propose using them for periodic metadata analysis. The method parses Lustre-specific on-disk structures and exposes them as SQL-queryable tables, requiring no intermediate files and placing no load on the metadata server. We implemented this method as a DuckDB extension and applied it to a Lustre deployment during its early access period. The analysis surfaces operationally relevant patterns: inode counts differ 4.9:1 across MDTs due to user behavior, 99.9997% of files use the default stripe configuration, the Python package manager uv created 69,785 cross-directory hard links on one MDT, and an automated process generated a 129,000-entry directory in under two minutes. We present these findings alongside the monitoring items they motivate, providing a reusable template for periodic metadata analysis. The tool is available at https://github.com/tsukuba-hpcs/lustre-query.
Sohei Koyama, Osamu Tatebe· IEEE International Symposium...· 0 citations
Persistent key-value stores (PKVS) are increasingly deployed in disaggregated settings that split compute, memory, and storage across separate server pools. This shift redraws the trust boundary: data that would remain within a single machine is now transported, cached, and rewritten across multiple hosts, expanding exposure to both network attackers and intra-infrastructure adversaries. This paper presents ANCHOR, a vision for end-to-end integrity and freshness in disaggregated PKVS. ANCHOR proposes a two-part semantics-aware architecture: 1) Persistence path: ANCHOR outlines encrypting and authenticating PKVS persistent files and preventing rollback with manifest versioning. 2) Volatile path: ANCHOR treats caches, indexes, and filters as untrusted hints unless accompanied by verifiable provenance, enforced by a TEE-resident policy. Finally, we outline key invariants and discuss enclave-friendly batching and asynchronous I/O to amortize verification without undermining disaggregation's performance and elasticity benefits.
Viraj Thakkar, Dongha Kim, Hokeun Kim et al.· 0 citations
A queueing network (QN) model of the rebalance process of the process used in MongoDB is developed, and a preliminary evaluation of performance prediction by the QN model via simulations is described, to understand the impact of different parameters of the rebalance process.
G. Paterakis, E. Ntallaris, K. Magoutis et al.· International Symposium on C...· 0 citations
