Semantics, Workflows, and Infrastructure: Understanding Agent Serving at Production Scale
A two-week trace of 11.7 million requests from a large-scale production platform for general-purpose agents, backed by inference infrastructure comprising over 10k GPUs is analyzed to characterize the platform at three connected levels: task-level initiation semantics, workflow-level execution patterns, and infrastruct...