Skip to content
Preprint

TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving

Oct 2026 · 1 citation · 51 references
Computer Science

TL;DR

TAO is presented, a tool-aware joint scheduling framework for LLM-based agentic serving that combines near-term tool readiness, node-specific cache credit with time-aware decay, and token footprint; a dynamic program jointly selects and places waiting sessions under capacity constraints.

Abstract

Agentic services alternate between model inference on clustered GPUs and external tools on distributed CPUs. In this scenario, existing systems (1) overlook tool-return readiness in service evaluation, wasting limited resources, and (2) use heuristic strategies to decide admission and placement on GPUs, resulting in suboptimal scheduling decisions. To bridge this gap, we present TAO, a tool-aware joint scheduling framework for LLM-based agentic serving. Its value interface combines near-term tool readiness, node-specific cache credit with time-aware decay, and token footprint; a dynamic program jointly selects and places waiting sessions under capacity constraints. Across the 12 reported model-workload-concurrency settings, TAO reduces latency by up to 26.8%, reduces mean time to first token (TTFT) by up to 36.3%, and increases decode throughput by up to 29.5% relative to the best evaluated state-of-the-art baseline for each metric.

View source

Similar papers

Preprint Nov 2026

Evolving Infra: Agent-Adaptive LLM Serving at Scale

Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload. These limi...

Hao Kang, Wei-Li Xu, Zi-Yang Li et al. · 0 citations
Sep 2026

PackServe: SLO-Aware Request Scheduling for Agentic LLM Serving at Scale

PackServe is a scheduler designed to reduce resource costs while meeting latency SLOs for agentic LLM serving, which uses compact white-box models to predict latency under prefill/decode interference and packs requests onto fewer serving instances while preserving KVC reuse and SLO constraints.

Zhi-Yuan Tan, De-Jiang Zhu, Jing-Zhe Jiang et al. · 0 citations
Book Open access Sep 2026

Congestion-Aware Serving of Agentic LLM Applications

CALM-MAS is proposed, a congestion-aware serving framework for LLM applications that treats LLM test-time computation as an elastic resource, dynamically adjusting the compute profile of admitted tasks to tame congestion.

Mouheb Ben Nasr, Muhammad Bilal, Alessandro Cornacchia et al. · 0 citations
Preprint Nov 2026

ToolSched: Resource-Efficient Tool Scheduling for Shared-Host LLM Agents

This work presents ToolSched, a Shell MCP Server for shared local hosts that serves multiple agents and schedules their operations, which improves resource utilization by 148.4% and end-to-end command throughput by 29.4%, demonstrating its effectiveness and practicality.

Shu-Cheng Zou, Yu-Hao Gu, Long Chen et al. · 0 citations
Sep 2026

PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Serving

It is shown that pipeline parallelism (PP), long overlooked because it offers little decode-latency advantage, can reduce JCT by providing a more favorable balance between prefill and decode efficiency, and PipeSwift is built, an optimized open-source pipeline-parallel runtime integrated with a tailored micro-batch par...

Shiju Wang, Fei Ren, Fang-Cheng Fu et al. · 0 citations
Preprint Oct 2026

Vesta: Queueing-Aware LLM Scheduling for Real-World Workloads

Agentic applications generate multi-turn sequences of LLM requests, with each new turn extending the context accumulated from prior interactions and tool calls. Prefix caching reuses the KV cache of earlier turns and avoids redundant prefill, but routing every turn to its cache-affine instance can create severe local q...

Lan-Cheng Zou, Bo-Wen Li, Wei Mao et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.