FlowCache: Workflow-Aware Intermediate-State Reuse for Multi-Tool LLM Serving

Abstract

Modern AI backend services have evolved from single-turn prompt completion into complex workflows that interleave retrieval, task planning, external tool invocation, and multi-step generation. Existing caching mechanisms—KV-cache reuse, RAG document caching, and model cold-start reduction—optimise individual pipeline components but leave substantial cross-step redundancy unexploited. We present FlowCache, a lightweight middleware that lifts caching to the workflow level by maintaining three semantic cache stores: a Plan Cache for reusable task-decomposition trees, a Tool Summary Cache for structured external-tool results, and a Prefix-KV Bundle Cache that packs system prompt, task template, and validated plan prefix into a single reusable KV state. A lightweight workflow signature matcher identifies reuse candidates, and a three-criterion correctness guard rejects stale or constraint-violating entries before serving. FlowCache is implemented as a transparent, OpenAI-compatible middleware layer on top of vLLM. Experiments on three backend-oriented workloads show that FlowCache reduces P50 end-to-end latency by up to 50% over the no-cache baseline and by up to 38% over a prefix-cache-only baseline, and reduces external tool calls per request by 38%, with end-task quality within 0.8 percentage points of the no-cache baseline and an observed stale reuse rate below 2% under our evaluation protocol.