Reinforcement Learning: A Comprehensive Review of Foundations, Algorithms, and Open Challenges

Abstract

Reinforcement learning (RL) is a branch of machine learning that studies how an agent should act within an environment to maximize cumulative reward. Supervised learning fits a function to labeled examples; RL instead learns through the consequences of trial-and-error interaction. It also differs from unsupervised learning, since its objective is to optimize a scalar reward signal rather than uncover latent structure. This review offers a self-contained, technically grounded survey for readers who understand probability and machine learning but may be encountering RL for the first time. We start from the Markov Decision Process (MDP) formalism and the Bellman equations that support nearly every RL algorithm. From there, we sort the main approaches into value-based methods, including Q-learning, Deep Q-Networks, and their variants; policy-based methods, such as REINFORCE, natural policy gradients, Trust Region Policy Optimization, and Proximal Policy Optimization; and actor-critic methods, which combine the two, including A2C/A3C, DDPG, TD3, and Soft Actor-Critic. For each group, we give the relevant equations, review representative empirical findings from the original publications, and examine sample efficiency, stability, and suitability for discrete and continuous action spaces. The discussion then turns to one of the field’s most consequential recent developments: applying RL to the alignment and reasoning abilities of large language models through Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and reasoning-focused methods such as Group Relative Policy Optimization (GRPO), used in DeepSeek-R1. We survey five persistent open problems: efficiency, training stability, scalability to high-dimensional spaces, safety and robustness, and generalization/transfer alongside the main mitigations proposed for each. We then turn to model-based RL, meta-learning, intrinsic motivation, and agentic, tool-using RL systems.