Hybrid Transformer–LSTM Forecasting and PPO-Based Intelligent Kubernetes Auto-Scaling Framework for Cloud Data Science Pipelines Using Predictive Workload Analytics and Multi-Objective Performance Optimization

Abstract

With the rise of cloud-native data science pipelines, dynamic infrastructure is needed to cope with the variable nature of AI/ML workloads, however, traditional auto-scalers based on Kubernetes use reactive threshold mechanisms leading to inefficient resource utilization and service level agreement (SLA) violations. In this paper, we introduce a novel prediction-based framework with an integrated intelligent auto-scaler based on a Hybrid Transformer-LSTM forecasting model and a Proximal Policy Optimization (PPO) reinforcement learning (RL) agent. The forecasting engine leverages historical time series data on performance metrics, such as CPU, memory, GPU utilization, and request latency, to provide accurate workload predictions. These predictions are used by the RL agent to determine optimal scaling decisions, pod placement and resource allocation strategies based on multiple objectives including minimizing latency, maximizing throughput, energy savings and lowering cloud expenditures. In extensive evaluations performed using multiple node clusters performing distributed deep learning, stream processing and real-time inference pipelines, the proposed solution outperforms conventional HPA/VPA and baseline ML scalers by offering improved accuracy, speed of scaling, reducing unnecessary allo-cations by 38% and meeting strict SLA requirements for bursty workloads.