Jul 2026· IEEE International Conference on Cloud Computing· pp. 315-326· 0 citations· 36 references
Abstract
Tiered memory environments that combine DRAM and CXL provide substantial scalability in modern systems, yet they introduce critical fairness challenges for containerized workloads. Existing Linux kernel memory management primarily relies on access-frequency-based page promotion and demotion to optimize system-wide performance. This approach causes two severe problems: (1) flaws in the current Cgroups memory-limit implementation can trigger infinite page reclamation loops in tiered memory systems, and (2) memory-intensive workloads monopolize fast-tier memory (e.g., DRAM), leading to the noisy neighbor problem in which tier occupancy becomes highly unfair even when containers have identical memory limits. This paper resolves the infinite page reclamation loop problem and proposes FATE, a fairness-aware memory allocation for multi-tenant tiered memory environments. FATE extends the Linux Cgroups memory controller to enforce per-tier high-watermark thresholds that are dynamically proportional to the share of total memory allocated to each container. For workloads requiring deterministic performance, FATE also supports configurable static fast-tier reservations. To enable precise enforcement, we introduce a tier-aware page counter that tracks per-NUMA-node, per-tier usage during allocation, migration, and reclamation. By strictly respecting fast-tier thresholds, FATE eliminates infinite reclamation loops and prevents unnecessary OOM killer invocation. Evaluation with diverse workloads in multi-container environments shows that FATE consistently achieves fair tiered memory allocation across containers with negligible runtime overhead. When occupancy ratios were quantified using Jain’s Fairness Index, FATE improved fairness by 7.3% to 31.7% over the baseline. More importantly, FATE significantly enhances performance predictability—a critical requirement for cloud service-level objectives—by providing strict isolation across memory tiers.
Results show up to 2.3x lower remote-access overhead compared to TCP-based designs, a 13% increase in cluster-wide memory utilization at rack scale, and up to 63% reduction in miss rate for memory-constrained tenants under skewed workloads.
Modern serverless platforms create function instances based on user-specified memory limits, which often result in memory over-provisioning even when the average runtime footprint is small. This over-provisioning reduces warm container density under a fixed memory budget, leading to early pool saturation, frequent evictions, and increased cold-start latency. To address this fundamental limitation, we propose Ditto, a system that integrates online memory rightsizing with value-aware container management. Ditto dynamically adjusts container memory to match actual usage, reclaiming unused memory while safely avoiding out-of-memory (OOM) conditions. In addition, it prioritizes containers based on their benefit per unit memory using a score-based eviction and keep-alive policy. We implement Ditto in Apache OpenWhisk and evaluate it across diverse workloads. Ditto reduces cold-start rate by up to 25.7% and latency by up to 33.3% compared to vanilla OpenWhisk, and by up to 14.8% and 19.4%, respectively, compared to RainbowCake.
Dongjae Lee, Kyuli Park, Yeonwoo Jeong et al.· IEEE International Conferenc...· 0 citations
The rapid adoption of large language models (LLMs) has increased the need for efficient multi-tenant inference systems that maximize GPU utilization. However, existing frameworks struggle to scale due to the high memory demands of model weights and key-value (KV) caches. We present DynamoServe, a multi-tenant LLM serving framework that addresses these challenges through three key innovations: (1) leveraging stranded GPU memory to offload model weights and KV caches, (2) mitigating resource fragmentation in multi-workload environments, and (3) improving memory locality through coordinated data placement and demand-driven weight migration across GPUs. Together, these techniques enable high-throughput, low-latency inference. Experiments on state-of-the-art models show that DynamoServe significantly improves memory efficiency without sacrificing latency.
Diman Zad Tootaghaj, Khaled Diab, Bob Lantz et al.· 0 citations
DRAM-based Processing-in-Memory (PIM) addresses the “memory wall” by executing computations directly inside main memory. However, memory interleaving and virtual memory limit contiguous data size visible to PIM units, constraining PIM task granularity. Fine-grained PIM tasks incur significant offloading overhead that negates PIM performance benefits. To mitigate this, existing PIM systems drastically isolate PIM memory or disable memory interleaving. These design choices, however, decrease the CPU memory bandwidth and introduce extra data transfer, leading to an additional “system memory wall” that degrades CPU performance and must be resolved to realize PIM’s full potential. In this work, we propose GUMPIM, a PIM system that allows interleaved CPU pages and non-interleaved PIM pages to coexist in a Unitary and Malleable memory space with Guaranteed PIM page allocation. GUMPIM enables zero-copy during PIM task offloading and maintains CPU memory bandwidth while ensuring low PIM offloading overhead. First, we propose a dual-track memory management mechanism consisting of independent page allocation and address translation for CPU and PIM pages. Second, we design GUMPIM interface hardware on PIM-enabled DRAMs to provide a dynamic address mapping for the different data layouts of CPU and PIM pages. Third, we propose a PIM-assisted page migration mechanism that transparently migrates pages while preserving CPU access bandwidth, thereby enabling guaranteed and accelerated PIM page allocation. GUMPIM requires no changes to commodity DRAM standards; all hardware modifications are limited to the DRAM side, ensuring full compatibility with existing CPUs and enabling immediate deployment on current HBMx- and LPDDRx-based PIM platforms. Our results show only <0.1% performance degradation for CPU workloads on GUMPIM, in contrast to the 25.8% degradation on PIM systems with memory interleaving turned off. For PIM workloads, GUMPIM reduces memory allocation and CPU-part computation times by 2.7× and 4.93×, respectively, yielding an end-to-end 2.3× speedup over a state-of-the-art baseline system.
Yilong Zhao, Fangxin Liu, Yiwei Hu et al.· ACM Transactions on Architec...· 0 citations
