Predicting Runtime in Spark‐Like Systems With Allocation‐Aware Deterministic Models

Abstract

Apache Spark is widely used for distributed data processing, but accurately predicting application execution time remains challenging because performance depends on application structure, resource configuration, and executor‐allocation behavior. This article presents two deterministic, graph‐based simulation models for predicting the execution time of Spark applications under static and dynamic executor allocation. The first model, the Static Allocation Model (SAM), represents an application as a monolithic directed acyclic graph (DAG) of stages and simulates execution using Spark's scheduling behavior under a fixed number of executor cores. To capture more realistic execution environments, this framework is extended to a Dynamic Allocation Model (DAM), which accounts for runtime changes in executor resources using backlog‐time and idle‐time thresholds. DAM dynamically adjusts executor allocation during execution and integrates these changes within the deterministic scheduling framework. Validation of SAM on three TPC‐DS benchmark queries—Query‐64, Query‐26, and Query‐52—yielded prediction errors of 2.9%, 4.3%, and 0.4%, respectively, with a mean absolute error of 2.5%. Experimental evaluation of DAM on TPC‐DS Query‐52 and the SparkBench K‐Means workload yielded prediction errors of 5% and 4.7%, respectively. To the best of our knowledge, this work presents the first deterministic approach for modeling Spark application execution time under dynamic executor allocation. The proposed models are lightweight, interpretable, and require minimal input data, offering a practical alternative for performance prediction and resource optimization in distributed data‐processing systems.