HYADES: Adaptive Consumer Tuning in Redis Streams via Hybrid Expert Selection

Abstract

Distributed messaging systems expose consumer-side parameters such as fetch batch size, blocking timeout, acknowledgment batching, and message reclamation thresholds that affect throughput, latency, and failure recovery. These parameters interact in non-obvious ways, making manual tuning difficult. Static configurations that perform well under one workload often degrade under dynamic traffic patterns.This paper presents HYADES, a hybrid control-plane architecture for adaptive consumer tuning. The system combines a reinforcement learning (RL) policy and a supervised learning (SL) model as competing experts, with a lightweight bandit-based selector choosing which expert generates the next configuration based on recent lag. Exploration decreases as lag increases, and a state-dependent safety projection constrains parameter values under high backlog.We evaluate HYADES on Redis Streams across workloads from 200 to 20,000 messages per second under three configurations (4 shards/2 consumers, 8 shards/4 consumers, 16 shards/8 consumers), including failure injection. Under heavy load at 8 shards/4 consumers, HYADES achieves the lowest average lag (313 messages) and reduces peak lag to 6,157 messages, compared to 9,126 for SL and 125,790 for RL. At 16 shards/8 consumers, it maintains the lowest peak lag (3,279 messages) while all methods converge to low average lag. The improvement is most pronounced during dynamic workload transitions, where HYADES adapts faster than single-strategy baselines. At 4 shards/2 consumers, performance is bounded by consumer capacity across all methods. Control-plane overhead remains low (5.99 ms average, 21.6 ms P99) relative to the 1 s control interval.