Dyserve compiles each workflow's per-node model and verifier choices in one integer linear program (ILP) over a heterogeneous backend pool, priced by skill-conditioned offline profiles that transfer across workflows.
Abstract
Agentic AI applications form an emerging serving workload in which a request creates a workflow: a directed acyclic graph of LLM and tool calls that exposes per-node model choices and optional quality operators such as verifiers. This workload falls between two existing layers. Model-serving engines execute individual calls efficiently but cannot see workflow structure, while agent frameworks fix the workflow but cannot see backend load, so neither jointly chooses each node's model, verifier, and backend under serving-time conditions. We present Dyserve, a workflow-aware serving layer that fills this gap. Dyserve compiles each workflow's per-node model and verifier choices in one integer linear program (ILP) over a heterogeneous backend pool, priced by skill-conditioned offline profiles that transfer across workflows. This couples with hardware entering only through per-model throughput sweeps, and is weighted to concentrate strong models and verification on the nodes whose errors propagate the furthest. Because no single latency-quality preference fits every workload mix, Dyserve pre-solves the program at several pressure levels at admission and shifts a workflow's uncommitted suffix among these strategies under load, keeping the solver off the load-shift path; a failed tool call triggers a one-time residual re-solve that preserves committed work.
Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available for batching concurrent requests. In multi-stage workflows, existing schedulers tend to prioritize either immediate prefix locality or overall workflow progress. However, under a shared KV cache budget, optimizing either objective in isolation can prolong tasklevel job completion time (JCT) through downstream delays or frequent prefix replacement. To strike a balance, we here propose TOPAS, a Task-Oriented Prefix-Aware Scheduler that jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution. TOPAS scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path against the near-term benefit of downstream prefix reuse, accounting for the costs of prefix movement and preemption. A task-level aging mechanism is also incorporated to prevent starvation. We implement TOPAS within the SGLang framework and assess its performance on three synthetic DAGs and two MetaGPT software-development workflows. Compared with the best performing baseline for each workload and metric, TOPAS reduces the mean/p99 JCT by up to 39.8%/49.4% on the synthetic workloads, while lowering mean JCT by 9.8% on MetaGPT-SOP and mean/p99 JCT by 22.0%/26.6% on MetaGPT-TL.
Hongqiu Ni, Han Tian, Chi Zhang et al.· 0 citations
Multi-agent applications increasingly rely on shared large language model backends in the public cloud, where bursty workloads cause requests from different agents to contend for the same LLM instances, leading to long queues, memory imbalance, and severe tail-latency inflation. Existing approaches typically prioritize requests using coarse workflow positions or static execution heuristics, which fail to adapt to short-term overload dynamics. We present FlowGuard, a workflow-aware overload controller for multi-agent LLM serving. Its key insight is that under sustained overload, GPU cycles spent on requests whose execution service-level-objectives (SLOs) are already violated are wasted. FlowGuard continuously recomputes per-request slack and prioritizes requests with the greatest remaining time before their deadlines, thereby maximizing on-time completions. In addition, a resource-aware dispatcher jointly accounts for KV-cache memory pressure and in-flight queue depth to reduce preemption across shared instances. Evaluated on a deliberately over-subscribed two-GPU backend, where all policies exhibit high absolute miss rates (i.e., the percentage of workflows that miss their deadlines), FlowGuard reduces the miss rate by 14–28% points over workflow-oblivious and static-priority baselines under BurstGPT-driven load, and by 34–38% points under co-located mixed-agent workloads.
Ali Zafar Sadiq, Haiying Shen· International Conference on...· 0 citations
Public LLM services serve diverse multi-agent applications with varying workflow dependencies and performance requirements. Requests generated by these applications often exhibit commonality and interdependence, yet current systems largely ignore such application-level structure. As a result, at the LLM engine cluster level, assigning requests to engines with the shortest queue can cause inefficient KV-cache transfers across GPUs. Using three representative multi-agent applications, we show that current scheduling methods miss opportunities to (a) improve performance through KV-cache reuse and reduced data transfer, and (b) increase goodput via batch management informed by workflow dependencies. Motivated by these observations, we propose a Workflow-Aware Scheduling system for Multi-Agent LLM systems (WaSMa) that incorporates cluster-and engine-level scheduling to optimize LLM request execution across GPU resources. Experimental results show that WaSMa reduces the P95 tail latency by up to 48% and improves goodput by up to 107% compared to existing methods.
Uttam Rao, Ali Zafar Sadiq, Haiying Shen et al.· International Conference on...· 0 citations
Deploying Large Language Models (LLMs) over the edge-cloud continuum faces severe stability challenges due to the conflict between stochastic network topology and complex workflow dependencies. Existing schedulers, relying either on computationally prohibitive Graph Neural Networks (GNNs) or topology-agnostic heuristics, fail to reconcile this tension. To bridge these gaps, we propose STEM, a service-level and topology-aware orchestration framework that formulates distributed LLM serving as a workflow-aware routing problem over a monitored service overlay, in which heterogeneous service instances act as specialized experts. At the core of STEM lies the STAR-PPO algorithm, utilizing a lightweight graph-free perception mechanism. By leveraging Squeeze-and-Excitation attention, it extracts critical bottleneck features from raw telemetry with linear complexity, bypassing the scalability limits of message-passing paradigms. To further achieve Pareto-efficient trade-offs, we develop a Dynamic Weight Adaptation (DWA) mechanism that autonomously recalibrates optimization preferences based on entropy-regularized metric drift. Extensive experiments on real-world datasets spanning 2,000 nodes demonstrate that our framework significantly outperforms state-of-the-art baselines. Specifically, STAR-PPO reduces network transmission costs by 96.8% and improves comprehensive inference efficiency by 24.4%, while sustaining robust zero-shot generalization across regions, with average latency within $1.09\times $ of a target-domain-retrained reference under a strict cross-region protocol. Code and data are available at https://github.com/gymorsiback/STARPPO
Yan Gao, Shaoyuan Huang, Yonghui Ye et al.· IEEE Transactions on Cogniti...· 0 citations
Modern data platforms rely on pipeline-oriented architectures that are rigid, hard to adapt, and lack native auditability. We present Agentic Data Services, a control-planedriven architecture for Big Data as a Service (BDaaS) that models workflows as adaptive, policy-aware service entities rather than static directed acyclic graphs (DAGs). The architecture combines (i) a dual-record execution model adapted from pharmaceutical batch manufacturing-versioned Master Batch Records (MBRs) for workflow definition and immutable Electronic Batch Records (EBRs) for execution traces-and (ii) workflow-level semantic caching that reuses results across semantically similar requests. We implement the system as Agentic DataHub, a set of Rustbased microservices deployed on Kubernetes, and evaluate the semantic-caching component on a reproducible benchmark using sentence-transformer embeddings (all-MiniLM-L6-v2) and a FAISS flat inner-product index. For clustered workloads-semantically related requests grouped into 10 clusters with 70% intra-cluster similarity-the cache reduces backend requests by 81% and median latency by 93%, with 40% P95 latency reduction. We discuss generalization across domains and the architectural constraints that bound these results.
Alexander Chernov· International Conference on...· 0 citations
Enterprises increasingly distribute computing workloads across multiple public and private cloud providers to reduce cost, avoid vendor lock-in, and improve resilience, but this multiplies the complexity of deciding, for every incoming task, which provider to use. Static or single-objective heuristics — always choosing the cheapest or always the fastest provider — routinely fail because cost, latency, and reliability trade off against one another in ways that shift with demand and provider conditions. This paper proposes and evaluates an explainable agentic AI framework for multi-cloud task allocation built on a contextual-bandit agent (LinUCB) that observes each provider's current price, estimated latency, and load before autonomously selecting a placement, then updates its policy online from the resulting cost, latency, and service-level-agreement (SLA) outcome. Because no public multi-cloud trace exposes simultaneous, ground-truth price/latency/capacity data across providers, the framework is evaluated on a controlled, fully documented discrete-time simulation of four heterogeneous providers under realistic load dynamics — a standard and disclosed methodology in this research area. Across 30 independent simulation runs of 3,000 tasks each, the agent achieved a statistically significant improvement over the strongest single fixedweight heuristic baseline (Static-Weighted) on every safety- and balance-related metric: 86.2% fewer SLA violations, 20.9% lower average latency, and 19.6% higher load-balancing fairness (Jain's index = 0.955 vs. 0.799, paired t-test, all p < 0.001), at a 20.6% higher cost. Under a simulated transient provider degradation (a 5× latency spike on one provider for 20% of a run), the agent held SLA violations to 0.4%, versus 9.9% for naive round-robin routing and 40.2% for cost-only routing, while remaining markedly cheaper than a purely latencyreactive baseline. To support the "explainable" requirement of agentic systems intended for production use, the framework exposes two complementary explanation layers: the bandit's own per-provider linear coefficients, and a surrogate Random Forest trained to imitate the agent's decisions (99.6% fidelity), whose permutation importance identifies observed latency and price as the dominant drivers of every allocation decision. These results indicate that a lightweight, interpretable contextual-bandit agent can deliver a favourable, auditable balance of cost, latency, SLA compliance, and fairness in multi-cloud environments, including under operational stress, without the opacity of deeper reinforcement learning or black-box agentic architectures.
Dr. Sajitha A V· International Journal of Int...· 0 citations