FlowGuard: Slack-Aware Overload Control for Multi-Agent LLM Serving

Abstract

Multi-agent applications increasingly rely on shared large language model backends in the public cloud, where bursty workloads cause requests from different agents to contend for the same LLM instances, leading to long queues, memory imbalance, and severe tail-latency inflation. Existing approaches typically prioritize requests using coarse workflow positions or static execution heuristics, which fail to adapt to short-term overload dynamics. We present FlowGuard, a workflow-aware overload controller for multi-agent LLM serving. Its key insight is that under sustained overload, GPU cycles spent on requests whose execution service-level-objectives (SLOs) are already violated are wasted. FlowGuard continuously recomputes per-request slack and prioritizes requests with the greatest remaining time before their deadlines, thereby maximizing on-time completions. In addition, a resource-aware dispatcher jointly accounts for KV-cache memory pressure and in-flight queue depth to reduce preemption across shared instances. Evaluated on a deliberately over-subscribed two-GPU backend, where all policies exhibit high absolute miss rates (i.e., the percentage of workflows that miss their deadlines), FlowGuard reduces the miss rate by 14–28% points over workflow-oblivious and static-priority baselines under BurstGPT-driven load, and by 34–38% points under co-located mixed-agent workloads.