Jul 2026· IEEE International Conference on Cloud Computing· pp. 246-256· 0 citations· 41 references
Abstract
Modern serverless platforms create function instances based on user-specified memory limits, which often result in memory over-provisioning even when the average runtime footprint is small. This over-provisioning reduces warm container density under a fixed memory budget, leading to early pool saturation, frequent evictions, and increased cold-start latency. To address this fundamental limitation, we propose Ditto, a system that integrates online memory rightsizing with value-aware container management. Ditto dynamically adjusts container memory to match actual usage, reclaiming unused memory while safely avoiding out-of-memory (OOM) conditions. In addition, it prioritizes containers based on their benefit per unit memory using a score-based eviction and keep-alive policy. We implement Ditto in Apache OpenWhisk and evaluate it across diverse workloads. Ditto reduces cold-start rate by up to 25.7% and latency by up to 33.3% compared to vanilla OpenWhisk, and by up to 14.8% and 19.4%, respectively, compared to RainbowCake.
Log-structured merge-tree (LSM-tree) key-value stores rely on caching to mitigate multi-component lookups and long tails, yet block and KP caches are prone to compactioninduced expiry, and all three cache types suffer from scan pollution under LRU eviction. We present AutoThermKV, a two-level in-memory architecture that decouples fast admission from hot-item protection. The front tier (HotFilter) uses a 64 Baligned bitmap/key/value page layout with Quick Space Reservation for near-constant admission cost; the back tier (HotCommitted) retains proven hot items via a lightweight LRU and a secondtouch promotion rule. Implemented atop RocksDB and evaluated on YCSB-like workloads with controllable temporal locality, AutoThermKV achieves up to 2.58 × throughput speedup, 2.60 × average-latency reduction, and tail-latency improvements of up to 20 × (p99) and 36 (p999) under modest memory budgets.
Yunfan Chi, E. Sha, Longshan Xu et al.· IEEE International Conferenc...· 0 citations
Cold start latency is a persistent, largely unsolved problem in serverless computing. When a function hasn't run recently, the platform must boot a VM or container, initialize a runtime, load the application framework, and only then handle the request — a process that takes 100ms to several seconds depending on platform and runtime. For latency-sensitive workloads, that overhead disqualifies the platform. We examine the root causes, build a decomposition model, and evaluate three mitigation strategies: memory snapshotting, fork-based execution (REAP), and Firecracker snapshot restore. We benchmark all three across five production-representative workloads. Snapshot restore cuts median cold start from 145ms to 28ms. Fork-based models (REAP) push that to 12ms. We also derive a warm pool sizing formula and model copy-on-write memory behavior under concurrent load. Our results show that fork-based approaches lead on latency while snapshot restore offers stronger isolation — making the right choice workload-dependent.
Satish Chavali· Journal of Scientific Engine...· 0 citations
To mitigate the cold-start overhead of serverless functions, two orthogonal approaches have been studied: in-memory caching and snapshot-based in-storage caching. In this work, we propose FaaSlim, a snapshot-based serverless computing system that integrates both approaches to reduce cold start latency. FaaSlim classifies VM pages into three categories, read, write, and permission update, based on page fault types, enabling partial caching of only a selected subset of pages rather than the entire VM while reclaiming the rest. When the VM is reused, only reclaimed pages are fetched from disk, reducing page fault overhead and function latency, while requiring less memory during idle periods. We also devise a partial caching-aware eviction policy, GDSF-CE, which selects VMs and page subsets to evict based on cache efficiency, a metric that quantifies the benefit of partial caching by relating latency reduction to memory consumption. Our evaluation using real-world traces shows that FaaSlim reduces the total overhead of snapshot-based cold starts by 23.0–27.8% compared to the state-of-the-art combination of in-storage and in-memory caching, FaaSnap with CIDRE’s GDSF-C, across diverse serverless workloads.
Sanghyeon Eom, Chanyoung Park, Gihong Lee et al.· International Conference on...· 0 citations
This work presents Oneiros, a dynamic remapping engine for multi-tenant LLM serving that dynamically repurposes GPU memory allocated for model parameters as KV cache capacity, enabling nonblocking, unidirectional parameter transfer.
Results show that object-value signals rank what to retain, while persistent responsibility determines which group bears reclamation pressure, which shows that object-value signals rank what to retain, while persistent responsibility determines which group bears reclamation pressure.
Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2\%--72.3\% and improves SLA-constrained throughput by up to 1.37$\times$. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2$\times$, while reducing the per-policy integration footprint from 500+ lines to under 50.
Yuanhang Gao, Xiangrui Yang, Yuanfeng Chen et al.· 0 citations
