Adaptive Inference Batching using Policy Gradients

Scheduling Mixed RL Rollouts Beyond Prefix Locality

Modern reinforcement learning (RL) post-training pipelines for large language models (LLMs) increasingly combine rollout workloads across multiple domains and feedback paradigms. Prefix-aware routing improves inference efficiency through cache reuse and load balancing, but it does not control how heterogeneous rollout sessions compete for KV-cache capacity. When reinforcement learning with verifiable rewards (RLVR), reinforcement learning from human feedback (RLHF), and agentic rollouts share an asynchronous inference service, their distinct sequence structures, interaction patterns, and KV-residency times create substantially different serving demands. Rollout scheduling must account for this heterogeneity without distorting the workload mixture specified by the trainer. We present MISA-T, a routing-layer admission policy for mixed rollout serving. MISA-T combines adaptive session admission, workload-aware KV-capacity allocation, and residency-time-aware KV accounting. In rollout-only ablations on Step3.7 and Qwen3.6-35B-A3B, MISA-T improves rollout throughput over a sweep-tuned cache-aware vLLM Router by 53.3% and 43.6%, respectively, while maintaining high prefix-cache hit rates. In a matched 50-iteration Step3.7 experiment, it increases rollout throughput by 35.6% and reduces mean iteration time by 22.8%, while keeping the consumed workload mixture close to the trainer target and achieving comparable task scores.

LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm

Large Language Models (LLMs) such as ChatGPT and Claude are widely used for information retrieval and problem-solving. Recent work has focused on improving scheduling algorithms to boost throughput while maintaining low latency. However, these approaches often assume Poisson request arrivals with constant rates - an assumption that fails to reflect the inherently bursty and dynamic nature of real-world traffic. We propose a lightweight extension to the state-of-the-art WAIT algorithm [1], which adapts to time-varying arrival rates without prior traffic knowledge. The proposed algorithm performs online estimation of request intensity based on observed interarrival times. Using Markov Modulated Poisson Process (MMPP)-based synthetic workloads with diverse request types, we conduct a simulation-based evaluation demonstrating that the proposed method achieves higher throughput than Sarathi-Serve [2], ORCA [3], and vLLM [4] in the evaluated low arrival-rate shift scenarios while maintaining comparable latency.

Refined Thompson Learning for Adaptive Bandits: Power-Efficient Flexibility Scheduling Across Data Centers

The rapid growth of large-scale AI workloads in data centers has placed increasing pressure on power grids in recent years. Since power systems must continuously balance supply and demand, there is growing interests in leveraging data-center workload flexibility as a grid service. We propose a contextual restless multi-armed bandit (CRMAB) framework in which a grid operator requests load reductions without observing internal job-scheduling decisions. Under index-ability guarantee, each data center or physical machine is modeled as a Markov decision process (MDP) over a cyclic virtual-machine (VM) job queue, with unknown rewards and transition dynamics learned online using Thompson sampling and Whittle-index policies. To improve learning under sparse and noisy observations, the framework augments an adaptive Thompson--Whittle (TW) policy with domain-informed transition priors and gated prior mixing. In baseline experiments, the best adaptive refined variant achieves 91.4\% of the oracle reward after 100 rounds and 96.8\% after 1,000 rounds. Across a 16-setting stress test spanning different state-space sizes and levels of contextual noise, the best refined variant consistently outperforms the original TW policy with high confidence while remaining competitive with EXP4. A graph-based prior further incorporates data-center hardware constraints, including computing-resource limits. Overall, the results demonstrate the economic potential of data-center flexibility as a grid service and highlight the importance of high-quality, open-source AI workload traces for developing and evaluating such services.

HYADES: Adaptive Consumer Tuning in Redis Streams via Hybrid Expert Selection

Distributed messaging systems expose consumer-side parameters such as fetch batch size, blocking timeout, acknowledgment batching, and message reclamation thresholds that affect throughput, latency, and failure recovery. These parameters interact in non-obvious ways, making manual tuning difficult. Static configurations that perform well under one workload often degrade under dynamic traffic patterns.This paper presents HYADES, a hybrid control-plane architecture for adaptive consumer tuning. The system combines a reinforcement learning (RL) policy and a supervised learning (SL) model as competing experts, with a lightweight bandit-based selector choosing which expert generates the next configuration based on recent lag. Exploration decreases as lag increases, and a state-dependent safety projection constrains parameter values under high backlog.We evaluate HYADES on Redis Streams across workloads from 200 to 20,000 messages per second under three configurations (4 shards/2 consumers, 8 shards/4 consumers, 16 shards/8 consumers), including failure injection. Under heavy load at 8 shards/4 consumers, HYADES achieves the lowest average lag (313 messages) and reduces peak lag to 6,157 messages, compared to 9,126 for SL and 125,790 for RL. At 16 shards/8 consumers, it maintains the lowest peak lag (3,279 messages) while all methods converge to low average lag. The improvement is most pronounced during dynamic workload transitions, where HYADES adapts faster than single-strategy baselines. At 4 shards/2 consumers, performance is bounded by consumer capacity across all methods. Control-plane overhead remains low (5.99 ms average, 21.6 ms P99) relative to the 1 s control interval.

WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

This work proposes WAR, a workload-aware rollout system that substantially accelerates synchronous agentic RL by jointly optimizing decoding and scheduling, and provides a practical path toward scalable long-context agent training.

SplitScaling: Adaptive Scaling for Disaggregated LLM Serving Against Traffic Bursts via DRL