CloudSafe-TP: Hybrid AI-Based SLO-Aware Predictive Thread-Pool Control for Distributed Cloud Backends
TL;DR
Overall, CloudSafe-TP provides predictive and uncertainty-aware pool-size adjustment that improves P99 SLO protection during bursty workloads while avoiding unnecessary concurrency and resource usage under moderate load.
Abstract
Cloud-based distributed services use backend thread-pool controllers to process changing request workloads. Fixed and reactive controllers may allocate too few threads during bursts or maintain unnecessary threads during normal operation. Existing schemes often depend on current measurements, fixed rules, or predictions without calibrated uncertainty, which may cause queue growth and P99 latency SLO violations. This paper presents CloudSafe-TP, a hybrid AI-based and SLO-aware predictive thread-pool control framework for distributed cloud backends. The novelty of CloudSafe-TP lies in combining a sparse Gaussian-process state-space model, conformal calibration, and model predictive control within an independent local controller that evaluates candidate pool sizes before they are applied. At each control interval, CloudSafe-TP predicts P99 latency, queue length, CPU usage, and memory usage over a short horizon, removes candidates that may violate the P99 SLO or resource limits, and selects a lower-cost safe configuration. The framework was evaluated on Oracle Cloud Infrastructure against analytical, neural-network-based, and reinforcement-learning controllers through five independent runs using random seeds 0–4. Under the bursty workload, CloudSafe-TP reduced mean P99 latency by 7.17–23.01%, queue length by 39.74–55.01%, and SLO violation rate by 77.11–89.57% compared with the baselines. Under moderate load, it used 20.70–26.60% fewer threads, reduced CPU usage by 3.90–7.64%, and reduced heap usage by 2.68–23.01%, while maintaining a P99 latency comparable to that of the strongest baseline. Paired t-tests further confirmed statistically significant improvements in P99 latency, queue length, and SLO violation rate under the bursty workload. Overall, CloudSafe-TP provides predictive and uncertainty-aware pool-size adjustment that improves P99 SLO protection during bursty workloads while avoiding unnecessary concurrency and resource usage under moderate load.
