This project explores the Countdown arithmetic reasoning task: given a set of numbers, produce an arithmetic expression that evaluates to a target value on the Qwen 2.5-0.5B base model and proposes two complementary extensions targeting these failure modes.
The novelty is twofold: tying curriculum design to the non-degeneracy of the RLOO advantage, and a zero-overhead adaptive sampler that discovers difficulty from reward rather than from a hand-set proxy.
This project asks a focused question: does ordering training data by difficulty make RLOO fine-tuning more effective on a reasoning task, and does a performance-adaptive curriculum outperform a fixed one?
These extensions focus on two approaches: curriculum learning and inference-time iterative feedback, where a Qwen-2.5-7B-Instruct model is used as a critic to provide corrective advice on previous round’s incorrect answers, allowing the model to learn from its own mistakes and make revisions.
This project investigates whether curriculum-based sampling can improve RLOO training by concentrating updates on prompts that are expected to be most informative and suggests a fundamental tradeoff between concentrating updates on potentially informative prompts and maintaining broad coverage of the training distribution.
This project studies whether curriculum-based prompt ordering can make RL fine-tuning more stable and sample-efficient for language-model reasoning, and when curriculum structure helps, when it fails, and what failure modes appear in small-scale online RL fine-tuning.