RL without Tears: An Introduction in the Era of LLMs

Abstract

Reinforcement learning (RL) has become a powerful and versatile paradigm in the era of large language models (LLMs). It is widely used to align models with human preferences and improve their reasoning capabilities. More recently, with the rapid rise of LLM-based agents, RL has become increasingly important for enabling models to learn from experience and build more capable and adaptive intelligent systems. Despite its growing importance, RL remains difficult for many LLM researchers to approach. Traditional RL literature often explains key concepts through robotics and control problems, making them less intuitive in the context of LLM training. To bridge this gap, this paper provides a comprehensive introduction to RL through LLM training examples. We first introduce the fundamental concepts of RL and explain key algorithms through a concrete example of training an LLM, including policy gradients, advantage estimation, importance sampling, and reward modeling. We then discuss recent advances in RL for LLMs, including improved reward construction, efficient training, and policy optimization. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models. We show how RL can enhance reasoning, train agents through environment interaction, and optimize multimodal understanding and generation. Finally, we summarize promising future directions and commonly used systems and datasets. We hope this paper provides an accessible introduction to RL and helps LLM researchers better understand and apply RL techniques in modern foundation models. The paper and related resources are available on GitHub at https://github.com/NiuTrans/RL-without-Tears, and an online version is provided at https://niutrans.github.io/RL-without-Tears-site/.