The method lets an HPC site compare deployment strategies using both user-visible performance and scheduler impact, then select the fastest strategy within its own RPC-demand limit.
Abstract
Wide Nextflow fan-outs on shared Slurm clusters can submit tens of thousands of short tasks. Deployment settings route them through individual jobs, arrays, or nested schedulers inside enclosing allocations. These settings determine workflow turnaround and RPC volume, a shared cost that can degrade scheduler responsiveness. Existing comparisons evaluate whole workflow systems, while per-task queueing metrics cannot span architectures that dispatch inside existing allocations. We contribute a reproducible measurement protocol and benchmark harness. A clean-start clock begins before backend startup or allocation requests, placing architecturally different backends on a common time axis. Per-user Slurm sdiag counters attribute request count as the primary RPC demand measure and controller processing time as sensitivity context, separate from cluster-wide state. We apply the method to Slurm native dispatch, Slurm job arrays, HyperQueue, and Flux on the shared ASU Phoenix production cluster and a single-user Dev cluster. On Phoenix, every aggregation strategy improves both objectives relative to native dispatch; Flux has the lowest RPC demand, while HyperQueue's fastest median walltime is not stable across replicates. On Dev, arrays and Flux improve walltime, while HyperQueue trades the lowest RPC demand for the slowest completion. The method lets an HPC site compare deployment strategies using both user-visible performance and scheduler impact, then select the fastest strategy within its own RPC-demand limit.
The default scheduler of Kubernetes, the state-of-the-art container orchestrator, uses fast, local placement decisions. Unfortunately, this design leads to resource fragmentation, reduced cluster usage, and overprovisioning. External solvers can compute global placement plans, but enforcing these plans in upstream clusters is hard. Kubernetes provides no native cross-node preemption, uncoordinated concurrent scheduling leads to inconsistencies, and replacing the default scheduler would sever deployments from upstream cycles. We present OPSche, an open-source Kubernetes Scheduling Framework plugin where external solvers can drive cluster-wide placement decisions in concert with the default scheduler. OPSche atomically validates and enforces solver-produced plans through coordinated framework hooks and supports three trigger modes: scheduling-failure, periodic, and stable-queue -- resp. triggered when a workload cannot be placed, at fixed time intervals, when the set of pending workloads stabilises. Each mode has a blocking variant for a finer tuning of placement quality, latency, and disruption. We pair OPSche with a constraint-based optimisation solver, showing its feasibility across a broad set of cluster configurations and reporting improvements of resource usage by up to 3.0% and scheduling latency by more than a second.
Henrik Christensen, S. Giallorenzo, J. Mauro· 0 citations
Load balancers in practice often rely on fixed heuristics such as weighted round-robin (WRR) or least connection (LC). Although these methods scale well, they do not capture differences in backend service capacity or runtime performance variations. which can increase tail latency and request drop rates in shared clusters. This issue becomes particularly significant for modern cloud inference workloads such as large language model (LLM) serving, where request processing times and backend utilization can vary substantially. To address this issue, we propose LP-WRR, an external control-plane mechanism that adjusts weighted round-robin weights using latency feedback. Our method periodically solves a constrained linear program (LP) to reduce expected service latency, then smooths the resulting weights with an exponential moving average so that updates do not fluctuate excessively, all while keeping the optimization logic off the data path. We then evaluate LP-WRR in a simulator for shared latency-sensitive services, using workloads derived from seven days of Azure ChatGPT traces as a representative inference workload, and compare it with the original load-balancing algorithm. Across the heterogeneity scenarios at low workload, our method improves both mean and tail latency, reducing p95 latency by $\mathbf{2 4. 7 \%}$ on average compared to baseline policies. Under the highest tested load, it also lowers the system-wide drop ratio in both homogeneous and heterogeneous infrastructure.
Hai Pham Thanh, Dang Hoang Nguyen, Anh Nguyen Tuan et al.· IEEE International Conferenc...· 0 citations
Accelerator clusters generate bursty network traffic: distributed training exchanges gradients via all-reduce, inference pipelines fan out requests, and storage competes with latency-sensitive RPCs. Poor placement amplifies congestion: a mis-scheduled training job can raise ECN mark rates by 3× and inflate co-located service tail latencies by 40%. Yet schedulers ignore real-time fabric state. We present Aegis, a scheduler that adapts placement using live fabric telemetry under operator-defined contracts. Aegis ingests per-ToR queue depths and ECN counters at 1-second granularity, incorporates a congestion-pressure term into placement scoring, and evolves online via a conservative bandit with exposure control. Contracts on tail latency, fairness, and congestion compile into safety envelopes that gate rollouts and trigger automatic rollback. We evaluate Aegis on an 80-node cluster (160 A100-80GB GPUs, 160 TPU v4 chips, 2-tier Clos fabric). Under production-derived workloads, Aegis reduces service p99 RPC latency by 27.3% (152.4 ms vs. 209.6 ms), cuts SLO violations by 3.6×, and lowers ECN mark rate by 51% while improving utilization from 68.4% to 79.1%. These gains hold under workload shifts and telemetry delays up to 60 seconds. The design is scoped to shared clusters where placement can materially reshape fabric contention.
Rui Li, Shuang Cao· Proceedings of the ACM SIGCO...· 0 citations
Multi-agent applications increasingly rely on shared large language model backends in the public cloud, where bursty workloads cause requests from different agents to contend for the same LLM instances, leading to long queues, memory imbalance, and severe tail-latency inflation. Existing approaches typically prioritize requests using coarse workflow positions or static execution heuristics, which fail to adapt to short-term overload dynamics. We present FlowGuard, a workflow-aware overload controller for multi-agent LLM serving. Its key insight is that under sustained overload, GPU cycles spent on requests whose execution service-level-objectives (SLOs) are already violated are wasted. FlowGuard continuously recomputes per-request slack and prioritizes requests with the greatest remaining time before their deadlines, thereby maximizing on-time completions. In addition, a resource-aware dispatcher jointly accounts for KV-cache memory pressure and in-flight queue depth to reduce preemption across shared instances. Evaluated on a deliberately over-subscribed two-GPU backend, where all policies exhibit high absolute miss rates (i.e., the percentage of workflows that miss their deadlines), FlowGuard reduces the miss rate by 14–28% points over workflow-oblivious and static-priority baselines under BurstGPT-driven load, and by 34–38% points under co-located mixed-agent workloads.
Ali Zafar Sadiq, Haiying Shen· International Conference on...· 0 citations
STORM is presented, a NIC-level scheduler for all types of RDMA workloads using NIC-only information: the known RDMA request size, and per-queue-pair backlog, and converts these signals into a small number of extra priority levels on the wire and prioritizes requests that are either near completion or blocking queued dependent work.
Jichun Wu, Ran Shu, Gianni Antichi et al.· Proceedings of the ACM SIGCO...· 0 citations
This study assesses the scalability of process-based and thread-based schedulers for many-core shared-memory systems using a memory-intensive row-wise quick-sort workload on large three-dimensional tensors. The process-based evaluation considers bounded prolific, bounded collective, and three pipe-based producer-consumer schedulers: one-to-one, one-to-many, and many-to-many. These pipe schedulers dynamically stream task identifiers to worker processes, exchanging increased inter-process communication overhead for enhanced runtime load balancing and flexible chunk-based task dispatching. The thread-based evaluation examines static, dynamic, guided, chunk-based, chunk-stealing, adaptive chunk, and AIMD adaptive scheduling strategies. The AIMD scheduler employs an additive-increase multiplicative-decrease policy inspired by TCP congestion control, utilizing an exponentially weighted moving average (EWMA) of CPU utilization to regulate a contention window that limits the number of concurrently active chunks. The adaptive chunk scheduler further modifies chunk size based on observed per-thread execution speed. Experimental results on a 24-core x86-64 platform indicate that thread schedulers deliver the highest overall performance, with dynamic and guided scheduling yielding the most favorable practical outcomes. Among process schedulers, pipe-based designs demonstrate the strongest scalability, with one-to-one pipes excelling for smaller workloads and many-to-many pipes preferred for larger workloads. In summary, lightweight thread scheduling is optimal for shared-memory row sorting, while AIMD/adaptive scheduling and pipe-based process scheduling remain valuable for contention-aware execution, explicit inter-process coordination, and distributed-style heterogeneous workload management.
Mejgan Dedaj, Argyro Gailla, T. Ioannou et al.· 0 citations