AI-Based Resource Scheduling in Distributed Data Engineering Platforms

Abstract

Modern distributed data engineering platforms process massive and diverse datasets, making efficient resource scheduling increasingly challenging. Traditional scheduling algorithms struggle to adapt to dynamic workloads and heterogeneous computing environments, resulting in poor resource utilization and increased execution time. This paper proposes an AI-Based Resource Scheduling Framework that integrates workload prediction, intelligent resource allocation, adaptive scheduling, and continuous performance monitoring. By leveraging machine learning and reinforcement learning, the framework dynamically optimizes scheduling decisions based on workload characteristics, resource availability, and real-time system feedback. Experimental results demonstrate improved resource utilization, reduced scheduling latency, enhanced scalability, lower operational costs, and better workload balancing compared to conventional scheduling approaches, making the framework well-suited for cloud-native and large-scale distributed data engineering environments.