CRAI: Calibrated Regret-Aware LLM Invocation for Efficient Cloud Resource Scheduling

Abstract

Large language models can contribute semantic and strategic reasoning to cloud schedulers, but invoking a large model at every scheduling epoch converts inference latency and monetary cost into a control-plane bottleneck. This paper studies the invocation decision itself. We present CRAI, a calibrated regret-aware invocation controller that treats an LLM as an expensive strategic meta-scheduler and calls it only when the estimated loss of reusing the cached policy is likely to exceed a budget-dependent dual price. CRAI combines a compact cluster-state representation, an online ridge predictor with an uncertainty bonus, and dual ascent that tracks a target invocation budget. The LLM-selected policy is never executed directly: a deterministic scheduler enforces CPU and memory feasibility while realizing one of four policy profiles—BALANCE, PACK, EDF, or LOCALITY. We evaluate the controller in a 48-node discrete-event simulator under normal, high-load, and bursty workloads using five independent seeds. Planner outputs are frozen across all runs to isolate invocation timing from provider-side nondeterminism. CRAI invokes the strategic planner for 9.54% of active epochs, a 90.46% reduction relative to always invoking, while average job completion time remains within 1.35% of the always-invoke reference. Against a static policy, CRAI reduces average JCT by 1.46% and SLA misses by 1.08 percentage points; against a state-change trigger, it reduces calls by 79.8% and JCT by 0.30 s. These results show that cloud schedulers can capture most of the value of strategic reasoning without paying its inference cost on routine epochs.