EnsembleLauncher is introduced, a recursively hierarchical workflow orchestrator for exascale systems, featuring a fully decentralized control plane and a programmable scheduling policy interface that outperforms state-of-the-art tools by more than four times.
Abstract
Scientific computing is increasingly shifting from monolithic applications to coupled simulation-AI workflows composed of highly heterogeneous tasks with diverse hardware, scale, and runtime requirements. As these workflows scale to leadership-class systems, the resulting extreme ensemble sizes and task variability can create orchestration bottlenecks. System-level schedulers are often configured for limited throughput, while workflow tools face scalability issues due to rigid control-plane topologies and static scheduling heuristics. We introduce EnsembleLauncher, a recursively hierarchical workflow orchestrator for exascale systems, featuring a fully decentralized control plane and a programmable scheduling policy interface. On the Aurora supercomputer, EnsembleLauncher successfully scales to the entire machine with up to eight million serial tasks, outperforming state-of-the-art tools by more than four times. Additionally, we implement a programmable scheduling interface and demonstrate a significant impact of scheduling policies on resource utilization for high-variance ensembles and active learning pipelines representative of modern coupled simulation-AI workflows.
High-Throughput Computing (HTC) environments tailored for high-concurrency resource efficiency require sophisticated orchestration to manage petabyte-scale data across heterogeneous resources. A critical but often overlooked challenge is workflow composition: the strategic grouping of tasksets within a Directed Acyclic Graph (DAG) to mitigate execution overhead while maximizing resource utilization. This paper presents a novel simulation framework for characterizing the interplay between taskset granularity and system-level constraints (e.g., job latency, failure rate, throughput, and I/O bandwidth). By exploring a high-dimensional parameter space, we quantify the performance sensitivity of diverse workflow topologies. Our results demonstrate that hybrid composition strategies, which dynamically balance taskset independence with execution grouping, can yield up to 3.8x throughput increase and a 14.9x reduction in network overhead. We further propose a multi-metric objective function that enables policy-driven optimization, allowing system architects to navigate the Pareto frontier between throughput, I/O cost, and CPU efficiency. These findings provide a rigorous foundation for automated workflow synthesis in distributed systems, offering a scalable model for next-generation scientific pipelines. All artifacts are publicly available.
As RISC-V processors are increasingly considered for embedded real-time and control-oriented systems, evaluating how timing behavior changes under increasing task concurrency becomes essential. Adding runnable tasks can amplify preemptions, context-switch activity, response-time variability, execution jitter, and deadline pressure. Existing RISC-V simulation and virtual-platform environments mainly target architectural exploration, functional validation, or full-system execution, and do not directly provide a controlled workflow for isolating scheduler-induced timing degradation across large configuration spaces. This paper presents nSim-RV, a configurable and reproducible RISC-V simulation and orchestration framework for scheduler-aware timing scalability evaluation. The framework combines automated campaign generation, deterministic workload configuration, structured dataset aggregation, duplicate validation, and timing-oriented metric extraction. The evaluation compares a standard shared-pipeline execution model with an nMPRA-inspired preserved-context mode under identical scheduler and workload conditions. The campaign includes CoreMark, Dhrystone, and a deterministic synthetic RT-Control workload, 2–32 concurrent tasks, 50 k–1 M cycle observation windows, cache-disabled and cache-enabled configurations, and four-stage and five-stage pipeline organizations, resulting in 864 validated configurations. Results show that increasing task concurrency amplifies timing variability and deadline pressure. Preserved-context execution reduces switching-induced disturbance and delays or reduces higher-pressure timing behavior in several trajectories. Under the five-stage cache-disabled RT-Control configuration at N = 32, it reduces the deadline miss ratio from 3.74% to 2.21%, corresponding to a 41.1% relative reduction, with the clearest benefits observed for Dhrystone and RT-Control at intermediate–high task counts.
Nicolai Iuga, N. Găitan, I. Zagan et al.· Computers· 0 citations
A comprehensive review of Kubernetes scheduling strategies published between January 2023 and January 2026 is presented and a multi-dimensional taxonomy is established that categorizes scheduling approaches based on common objectives, modification methods, optimization methodologies, targeted workloads, evaluation methods, scheduling scopes, and performance metrics.
Mohammed Alhakimi, R. Latip· Computers· 0 citations
The means to execute and orchestrate software components has changed from human-written code to descriptive prose. In high performance computing, this transition is represented in application orchestration, workload management, and system monitoring and debugging, to name a few. The underlying means to enable descriptive definition of tasks is the use of the Large Language Model with associated tool functions and resources. A combination of a model with access to such resources, modeled in software, encompasses an autonomous framework. As fully automated and agentic frameworks are developed for science, it is important to assess reliability and strategies scoped to specific tasks. In this work, we assess the extent to which an agentic framework can optimize and run an HPC scaling study with a low latency network in Amazon Web Services, accurately transform HPC job specifications between workload managers, and design and run an entire biosciences workflow. We find that the framework completes all three tasks while surfacing task-specific failure modes. In the scaling study, agents deploy and optimize applications but monitor running jobs inefficiently, preferring conservative fixed waits over event subscriptions. In job translation, they convert specifications between Slurm and Flux with high accuracy, with processor-affinity flags the most common error. In the bioscience workflow, the agent reproduces an expert-written variant-calling pipeline almost exactly -- agreeing with the reference call set in 18 of 19 completed runs -- and reaches this result through many distinct yet functionally equivalent workflow implementations. This information is invaluable moving forward to developing multi-cluster setups with scheduling and transformation handled by agents.
LLM agents increasingly solve tasks by invoking multiple tools, where parallel execution is essential for low latency but difficult to manage safely. Existing agent benchmarks primarily evaluate tool selection, argument generation, and end-to-end success under mostly serial execution, largely overlooking valid parallelization and resource-constrained scheduling. This missing scheduling dimension creates a practical failure mode: serial execution is safe but slow, while resource-agnostic parallel execution is fast but prone to avoidable resource overflows. To address this gap, we introduce PeakBench, a benchmark of executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles. A central challenge in evaluating such workflows is attribution: failures and inefficiencies may arise from incorrect dependency planning, poor resource-constrained scheduling, or both. PeakBench addresses this challenge with a two-part evaluation framework that disentangles logical planning from physical scheduling, with dedicated metrics for each dimension. Using this framework, we show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints. We further show that exposing resource information can reduce avoidable overflows and improve resource utilization, making PeakBench a useful testbed for diagnosing resource-aware agent behavior. Code is available at https://github.com/Czzzk/Staggering-the-Peaks.
Zhi-Kai Chen, Xu-Xiang Zhong, Song-Yan Li et al.· 0 citations
This work organizes agentic workflows in a taxonomy and presents its first architectural characterization with a production study at Microsoft Azure and a controlled study of open-source frameworks, showing that agentic execution is fragmented and heterogeneous.
Jirong Yang, Peizhe Liu, Chaojie Zhang et al.· 2 citations