Jun 2026· 2026 8th International Conference on Electronic Engineering and Informatics (EEI)· pp. 77-83· 0 citations· 18 references
Abstract
The massive memory footprint of the KV Cache is a critical bottleneck in scaling Large Language Model (LLM) inference to ultra-long contexts. Existing single-node or compression-based methods either compromise model fidelity or fail to scale efficiently. This paper introduces a four-tier heterogeneous storage architecture that extends the KV Cache across GPU VRAM (L1), host RAM (L2), local NVMe SSDs (L3), and an RDMA-enabled distributed fabric (L4). To mask cross-tier I/O latency, we propose the Predictive Multi-tier KV Scheduling (PMKS) algorithm, which exploits the sequential access patterns of autoregressive decoding to prefetch blocks asynchronously. We evaluate the system on an ODCC AI Storage Lab testbed using DeepSeek-R1 with context lengths up to 100K tokens. Compared to a vanilla vLLM baseline under batch-size-16 workloads, our architecture achieves a 90% reduction in Time-to-First-Token (TTFT) for warm-start scenarios and a 20-fold increase in token throughput, while maintaining stable decoding latency. Finally, a TCO analysis indicates that storage-augmented GDDR-based GPU nodes can deliver long-context throughput comparable to premium HBM-based infrastructures at a lower system cost, offering a cost-effective paradigm for enterprise-scale LLM deployment.
FLINT is proposed, a workload-driven HBF substrate for capacity-scalable LLM inference that integrates HBF as a memory-capacity tier alongside HBM while addressing three adoption challenges.
Geraldo F. Oliveira, Arash Tavakkol, Xiang-Yu Zhu et al.· 0 citations
This work proposes Turbo, a first-of-its-kind in-network aggregation system that accelerates long-context inference by offloading query broadcast and attention aggregation to switches and introduces a rolling forward scheme that propagates states to enable cross-stage updates.
Ying Wan, Yuchen Xu, Chuwen Zhang et al.· Proceedings of the ACM SIGCO...· 0 citations
The rapid adoption of large language models (LLMs) has increased the need for efficient multi-tenant inference systems that maximize GPU utilization. However, existing frameworks struggle to scale due to the high memory demands of model weights and key-value (KV) caches. We present DynamoServe, a multi-tenant LLM serving framework that addresses these challenges through three key innovations: (1) leveraging stranded GPU memory to offload model weights and KV caches, (2) mitigating resource fragmentation in multi-workload environments, and (3) improving memory locality through coordinated data placement and demand-driven weight migration across GPUs. Together, these techniques enable high-throughput, low-latency inference. Experiments on state-of-the-art models show that DynamoServe significantly improves memory efficiency without sacrificing latency.
Diman Zad Tootaghaj, Khaled Diab, Bob Lantz et al.· 0 citations
Log-structured merge-tree (LSM-tree) key-value stores rely on caching to mitigate multi-component lookups and long tails, yet block and KP caches are prone to compactioninduced expiry, and all three cache types suffer from scan pollution under LRU eviction. We present AutoThermKV, a two-level in-memory architecture that decouples fast admission from hot-item protection. The front tier (HotFilter) uses a 64 Baligned bitmap/key/value page layout with Quick Space Reservation for near-constant admission cost; the back tier (HotCommitted) retains proven hot items via a lightweight LRU and a secondtouch promotion rule. Implemented atop RocksDB and evaluated on YCSB-like workloads with controllable temporal locality, AutoThermKV achieves up to 2.58 × throughput speedup, 2.60 × average-latency reduction, and tail-latency improvements of up to 20 × (p99) and 36 (p999) under modest memory budgets.
Yunfan Chi, E. Sha, Longshan Xu et al.· IEEE International Conferenc...· 0 citations
OasisKV is presented, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding and observes that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD).
Can Xiao, Sukmin Cho, Junbong We et al.· 0 citations
Modern large language model (LLM) inference is increasingly dominated by memory-bound operations, making processing-near-memory with processing-in-memory (PNM-PIM) an attractive approach for accelerating the decode phase. However, recent long-context LLMs adopt interleaved local and global attention (ILGA), which introduces heterogeneous execution characteristics across transformer blocks. In realistic serving environments, this heterogeneity is further amplified by mixed-length requests, where inputs with widely different context lengths are processed concurrently. These trends break a key assumption underlying existing PNM-PIM systems that transformer blocks exhibit similar latency and resource demands and can be efficiently mapped to a uniform pipeline. Under ILGA and mixed-length workloads, this assumption no longer holds, leading to severe pipeline imbalance and low utilization in prior PNM-PIM designs. Moreover, PNM-PIM–only systems struggle to efficiently support long-context prefill, which remains compute-intensive and is better suited for GPU execution. In this paper, we propose Aurora, a GPU–PNM–PIM disaggregated system designed to efficiently serve mixed-length LLM inference under ILGA. Aurora introduces an ILGA-aware multi-path PNM-PIM pipeline that explicitly accounts for block-level heterogeneity and request-length diversity, improving pipeline utilization without overprovisioning tensor parallelism. Also, Aurora further adopts a stream-oriented Softmax design to reduce stage-level latency imbalance during decoding. To enable end-to-end inference under disaggregation, Aurora treats KV cache transfer as a pipeline operation and coordinates request issuance and scheduling to avoid transfer-induced stalls. Our experimental results show that Aurora achieves up to 8.5 × and 2.2 × higher throughput than the GPU-only baseline and prior PNM-PIM systems, respectively, in end-to-end Llama4-Scout inference, while requiring a comparable or smaller number of devices.
Hyeonu Kim, Seunghyuk Yu, Minjeoung Kim et al.· International Conference on...· 0 citations