A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing

OpScale: Operator-level Provisioning and Autoscaling for LLM Serving

OpScale is presented, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving that attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.

Co-Optimizing Request Scheduling and KV Caching for Edge LLM Serving

Deploying large language models (LLMs) on edge nodes enables low-latency and privacy-preserving inference, but faces severe resource constraints under high-concurrence workloads. While existing inference systems leverage intranode key–value (KV) caching to improve efficiency, they largely neglect the unique complexities of multinode edge environments. Specifically, reactive KV cache eviction policies suffer from temporal uncertainty, often discarding reusable KV caches prematurely, while the tight coupling between request scheduling and cache placement often leads to myopic decisions that exacerbate load imbalance and resource contention. To address these challenges, we propose a dynamic block-level paradigm that treats KV blocks as the fundamental units for caching and scheduling, enabling dynamic sharing, generation, and eviction of arbitrary-length prefixes. We present complete modeling of the spatiotemporal coupling between scheduling and caching under block-level granularity, capturing intricate interactions overlooked by prior work. Based on this model, we design an online joint optimization algorithm, which applies to general edge LLM serving scenarios. The algorithm decouples spatiotemporal dependencies via randomized rounding over per-slot subproblems, achieving a balance between real-time responsiveness and long-term system efficiency. Theoretical analysis establishes high-probability near-optimality guarantees, and extensive experiments show that our method reduces the average time to first token (TTFT) by up to 54.02% over existing baselines.

Scalable Caching with Amazon ElastiCache Redis Cluster Mode: A Quantitative Performance Study

Enterprise applications increasingly depend on distributed caching to sustain sub-millisecond response times at scale. Amazon ElastiCache Redis, operating in cluster mode, provides horizontal partitioning across configurable shard topologies, enabling throughput and memory capacity to grow in proportion to demand. While many organizations have adopted cluster configurations, empirical guidance on topology selection, key distribution optimization, and the measurable performance impact of individual tuning techniques remains sparse. This paper addresses that gap through systematic benchmarking across multiple cluster topologies (3 to 90 shards), three Graviton-based instance families (m6g, r6g, r7g), three workload profiles, and five optimization techniques, augmented by client library analysis, memory optimization guidance, and production cost validation. Production case studies from financial services, e-commerce, and real-time analytics platforms validate laboratory findings. Results offer empirical guidance for cloud architects designing caching architectures that balance latency requirements, horizontal scalability objectives, and infrastructure cost efficiency.

LP-WRR: Towards Adaptive Performance-Aware Load Balancing

Load balancers in practice often rely on fixed heuristics such as weighted round-robin (WRR) or least connection (LC). Although these methods scale well, they do not capture differences in backend service capacity or runtime performance variations. which can increase tail latency and request drop rates in shared clusters. This issue becomes particularly significant for modern cloud inference workloads such as large language model (LLM) serving, where request processing times and backend utilization can vary substantially. To address this issue, we propose LP-WRR, an external control-plane mechanism that adjusts weighted round-robin weights using latency feedback. Our method periodically solves a constrained linear program (LP) to reduce expected service latency, then smooths the resulting weights with an exponential moving average so that updates do not fluctuate excessively, all while keeping the optimization logic off the data path. We then evaluate LP-WRR in a simulator for shared latency-sensitive services, using workloads derived from seven days of Azure ChatGPT traces as a representative inference workload, and compare it with the original load-balancing algorithm. Across the heterogeneity scenarios at low workload, our method improves both mean and tail latency, reducing p95 latency by $\mathbf{2 4. 7 \%}$ on average compared to baseline policies. Under the highest tested load, it also lowers the system-wide drop ratio in both homogeneous and heterogeneous infrastructure.

From Per-Job Data to an Aggregated Workload Insight: A Toolkit for Profiling HTCondor Workload

HTCondor users for high-throughput computing often struggle to quickly understand how their computational workloads are performing. Current interfaces expose large volumes of raw job data, making it difficult to diagnose common issues such as jobs stuck on hold, poor resource utilization, or unexpected failures. These issues further snowball when dealing with large clusters. We present a Python toolkit, developed at the Center for High Throughput Computing (CHTC) at the University of Wisconsin–Madison, that bridges this gap. To be included as a part of the HTCondor suite, given a single cluster ID the toolkit aggregates raw job data into four cluster-level views: a status dashboard showing the distribution of job states, a runtime histogram revealing duration variance and flagging anomalously short runs, a hold classifier that groups held jobs by reason code with plain-language explanations, and a resource utilization report comparing requested versus actual CPU, memory, and disk usage. Each view is designed to answer a specific question a researcher or facilitator would ask when triaging a workload, transforming thousands of individual job records into a concise, interpretable summary.

Future Generation Computer Systems