TL;DR
RL-ICE is proposed as an innovative scheduler that can work in such a cloud continuum by leveraging a multi-cluster and hierarchical RL to satisfy both user Quality of Experience (QoE) metrics and tenant’s costs.
A comprehensive review of Kubernetes scheduling strategies published between January 2023 and January 2026 is presented and a multi-dimensional taxonomy is established that categorizes scheduling approaches based on common objectives, modification methods, optimization methodologies, targeted workloads, evaluation methods, scheduling scopes, and performance metrics.
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
OpScale is presented, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving that attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.
With the widespread adoption of LLM-based chatbots, cloud-dependent solutions have come to dominate the market. However, open-source pre-trained LLMs are enabling implementation of local solutions. Achieving competitive performance locally requires the ability to run high-parameter models. Here, the primary bottleneck is GPU VRAM capacity, which limits model parameter size. Furthermore, the efficiency of inference optimizations such as kv caching depends directly on the amount of available VRAM remaining after the model is loaded. In such a scenario with hardware constraints, we conduct user tests by applying kv cache quantization. As a result, we identify distinct performance trends in critical metrics such as Time-to-First-Token (TTFT) and Total Generation Time. Additionally, we evaluate model accuracy results using the LLM-as-a-judge paradigm.
This work introduces a novel LLM-based predictive scheduling system designed to enhance operational efficiency while reducing the environmental impact of data centers, using an LLM to predict key metrics such as execution time and energy consumption from source code.