Distribution-Value Coevolution for Adaptive RLHF Data Scheduling

Abstract

Reinforcement learning from human feedback (RLHF) has become the cornerstone of aligning large language models (LLMs) with human intent. Yet a fundamental question remains unaddressed: how should training data be scheduled when both the model's capabilities and the utility of data are constantly evolving? Current pipelines rely on fixed or uniform sampling, treating data value as static, an assumption we demonstrate to be fundamentally flawed. We identify and formalize the Distribution-Value Coevolution principle: the training value of data is not intrinsic, but emerges dynamically from the interaction between data characteristics and the model's evolving capability boundary. What is highly informative at one stage may become redundant, or even detrimental, at another. This insight demands a paradigm shift from static to adaptive curriculum design. We operationalize this principle through a unified framework with three components: (1) distribution-level organization that groups training data into coherent distributions; (2) sliding-window influence estimation that continuously tracks each distribution's evolving training value; and (3) bandit-guided scheduling that adaptively allocates resources with provable exploration-exploitation guarantees. Experiments show that this approach yields measurable improvements, with up to a 57.1% relative (or 8.9% absolute) improvement on AIME24 for Llama3.2-3B, and gains also observed for models ranging from 1B to 7B parameters.