This work proposes ScaleSense, a proactive, query-level resource scaling framework that addresses the critical performance-cost trade-off while maintaining low-overhead inference latency, and confirms its practical performance in production deployments.
Abstract
Cloud-native serverless data warehouses achieve fine-grained elasticity by decoupling storage from compute, yet determining the optimal resource allocation for highly heterogeneous ad-hoc queries remains a formidable industrial challenge. Our analysis of production workloads in Alibaba AnalyticDB exposes a costly ``provisioning trap'': the fear of catastrophic resource depletion drives users to blindly over-provision resources, wasting immense monetary budgets without alleviating non-CPU bottlenecks (e.g., I/O saturation). To break this impasse, we propose ScaleSense, a proactive, query-level resource scaling framework. Specifically, it features a multi-faceted query encoder that jointly models plan topologies and hardware specifications. Crucially, a quantile-based resource predictor estimates multi-dimensional physical footprints, acting as a reliable safety net for optimal resource scaling. An auto-scaling controller then navigates the performance-cost Pareto frontier, dynamically tailoring allocations to specific business priorities without requiring model retraining. Evaluations on over 1.36 million production queries show that ScaleSense achieves state-of-the-art prediction accuracy with good prediction interval coverage. By achieving a 76.7% relative improvement in optimal resource configuration selection over the best baseline, this approach addresses the critical performance-cost trade-off while maintaining low-overhead inference latency, confirming its practical performance in production deployments. Under the performance-optimization policy, ScaleSense satisfies user-defined performance requirements while reducing monetary cost by up to 5.22x.
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
PRISM, a prediction-guided runtime framework that jointly selects model variants and CPU allocations for containerized edge microservices, and adapts each pipeline stage in place and minimizes predicted CPU-package energy under deadline, resource, and offline model-level Quality of Result constraints is presented.
Uwe Gropengießer, Thomas Reuter, Dominik Schön et al.· 0 citations
Object-relational mapping (ORM) design remains largely driven by fixed heuristics that fail to capture workload-specific tradeoffs among query latency, insert cost, and memory footprint. We present Y-Map, a hybrid neural--symbolic framework for performance-aware ORM schema design that synthesizes valid schema candidates and predicts their performance without requiring workload execution at inference time. Y-Map leverages Alloy to enumerate correctness-preserving ORM schemas and ranks them using a multi-encoder regression model that fuses structural, syntactic, and semantic representations with compact schema-level features. By predicting continuous performance objectives---insert latency, query latency, and memory footprint---Y-Map enables Pareto-aware selection without per-candidate benchmarking during inference. We evaluate Y-Map on nine object models from e-commerce, banking, and healthcare. Relative to two representative baselines, Leant and DTS, Y-Map yields improved aggregate Pareto quality (Generational Distance and Hypervolume) while reducing inference time and memory overhead. The experimental results show that integrating symbolic validity guarantees with learned performance prediction provides a practical, scalable solution for workload-aware ORM optimization.
Serverless computing has emerged as an attractive deployment model for deep learning model inference workflows, enabling elastic scaling and fine-grained resource billing across function instances. However, scheduling in this setting introduces a competing-objective challenge: placement decisions simultaneously govern data transfer overhead, determined by whether dependent function instances are co-located, and model loading overhead, determined by whether required model weights are memory-resident on the target node. We present AnchorDL, a joint-cost look-ahead scheduler that minimizes the combined cost of both overheads at each placement decision, with a forward term that avoids greedy suboptimality across adjacent data dependencies. Evaluated against three baselines across chain, fan-in, and fan-out workflow topologies under trace-based workloads, AnchorDL achieves the lowest median end-to-end latency across all evaluated workflows and reduces P90 latency by up to 57.4% against the model-centric baseline in the fan-out workflow. The look-ahead term further contributes substantially beyond greedy joint-cost placement.
Min Chang Kim, Hyungjun Kim, Hokun Park et al.· IEEE International Conferenc...· 0 citations
This paper employs five lightweight machine learning models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution.
Reza Farahani, Zoha Azimi, Mario Colosi et al.· 0 citations
The rapid deployment of machine learning systems across cloud, edge, and enterprise environments has brought model optimization to the forefront of systems-engineering. Despite a rich literature spanning quantization, pruning, knowledge distillation, parameter-efficient fine-tuning (PEFT), and inference-time optimization, practitioners are often left navigating these techniques through heuristics rather than principled methodology. We argue that optimization should be formulated as a constraint-driven, multi-objective engineering decision and introduce a unified framework that characterizes any production deployment along five interacting constraint dimensions: data availability, latency budget, memory budget, accuracy tolerance, and retraining budget. Building on this taxonomy, we synthesize empirical gains reported across the research literature and map them to operational constraints rather than algorithmic categories. To ensure practical relevance, we selected these techniques by reviewing recent literature for methods that report measurable improvements against critical deployment bottlenecks. We propose a prescriptive decision framework and provide optimization pipelines for four representative industrial scenarios to illustrate it in practice. To the best of our knowledge, this work provides one of the first structured attempts to formalize model optimization as a constraint-aware, multi-objective engineering process, synthesizing quantitative evidence from the research literature.
Dhruv Shivkant, S. Mohanty, S. Rai et al.· 0 citations