Hot–Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving

Abstract

Large language model (LLM) serving is increasingly agentic, with multi-turn sessions that idle between actions yet must retain their full context. Limited GPU memory capacity forces inactive KV states to be evicted, so resuming a session incurs either costly recomputation or slow interconnect transfers. To address this, high bandwidth flash (HBF)—an on-package 3D-NAND memory offering orders-of-magnitude greater capacity than high bandwidth memory (HBM) at comparable read bandwidth—has emerged as a strong candidate. However, its high read energy and limited write endurance make it impractical to serve <italic>all</italic> KV traffic. Fortunately, our analysis shows that agentic KV states exhibit distinct access patterns: a small hot set is read for every decoding step, while a large cold pool is read only when a paused session resumes. Exploiting this, we place the hot set in HBM and the cold pool in HBF, forming a hot–cold KV hierarchy within the GPU memory tier. On agentic workloads with Qwen3-Coder-30B-A3B, our design delivers 14 ms time-between-tokens (TBT) and adds only <inline-formula><tex-math notation="LaTeX">$\approx \!0.1$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>≈</mml:mo><mml:mspace width="-0.166667em"/><mml:mn>0</mml:mn><mml:mo>.</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="kim-ieq1-3729099.gif"/></alternatives></inline-formula> ms of resume latency on top of prefill, while hosting <inline-formula><tex-math notation="LaTeX">$24\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>24</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="kim-ieq2-3729099.gif"/></alternatives></inline-formula> more concurrent sessions per GPU. By confining HBM to the hot set, our design also cuts read power by 7.6 kW per 8-GPU node relative to serving all KV from flash—establishing HBF as a cold-tier complement to HBM rather than its replacement.