A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulator
TL;DR
A comprehensive review of approximately 100 peer-reviewed studies that apply RL in the CARLA simulator shows that model-free RL overwhelmingly dominates the field, accounting for more than 80% of existing studies, with DQN, PPO, and SAC being the most frequently adopted algorithms.
Abstract
Reinforcement learning (RL) has become an increasingly important framework for autonomous driving, while the CARLA simulator has emerged as a leading benchmark environment for training and evaluating RL-based driving agents. Despite rapid growth in this area, the literature remains fragmented, making it difficult to identify prevailing methods, experimental practices, and open challenges. This paper presents a comprehensive review of approximately 100 peer-reviewed studies that apply RL in the CARLA simulator. The surveyed works are organized into major methodological categories, including model-free, model-based, hierarchical, hybrid, and other specialized RL approaches. Our analysis shows that model-free RL overwhelmingly dominates the field, accounting for more than 80% of existing studies, with DQN, PPO, and SAC being the most frequently adopted algorithms. We also examine how these studies formulate driving problems through different state representations, action spaces, and reward designs, and we summarize the evaluation landscape in terms of metrics, towns, scenarios, and traffic configurations. Finally, we highlight persistent research challenges such as sparse reward design, generalization, sim-to-real transfer, safety, and limited behavioral diversity, and we discuss emerging directions that may help address these limitations. This review provides a structured reference for researchers entering the field and offers a foundation for future advances in RL-based autonomous driving in CARLA.