AI Networking Cookbook: Practical recipes for AI-assisted network automation and development
TL;DR
HyMCache is presented, a KV-cache framework that integrates CXL-hybrid memory (CXL-HM) for multi-turn LLM serving that uses request-level prefix prefetching and opportunistic write buffering to stage latency-critical reads in device DRAM, enabling DRAM-scale KV-cache efficiency at SSD-level cost.
LM-CXD bridges the semantic gap between the serving engine and the device, and exposes NAND-to-DRAM progress to the serving engine, and uses device DRAM as a GPU-accessible buffer to hide NAND latency under limited device DRAM.
TierKV is presented, a mobile LLM inference framework built on Predictive Multi-Tier Cache Optimization (PMCO), which improves prefill throughput by up to 17.6x over existing mobile LLM frameworks, reduces RAM-resident KV cache by 12.5-34%, thereby enabling substantially longer contexts under the same memory budget, wh...
This model reveals one key opportunity: dividing a restore request proportionally between the storage path and the GPU can improve inference performance while still meeting SLOs, and reduces the KV-cache storage stack to a performance model based on per-tier capacity, per-tier and interconnect bandwidth, and GPU arithm...
OasisKV is presented, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding and observes that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD).
It is found that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth, not only on device bandwidth.
A DRA driver is presented that makes composable CXL memory a schedulable cluster resource, and the resulting shared-memory tier for cross-node KV-cache reuse in LLM serving is evaluated, demonstrating memory disaggregation rather than prefill/decode disaggregation.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.