Towards Adaptive and Retriever-friendly Retrieval-augmented Generation via Reinforcement Learning

Abstract

Retrieval-augmented Generation (RAG) has emerged as an effective paradigm for enhancing large language models (LLMs) with external knowledge, delivering substantial performance gains without costly parameter updates, particularly in low-resource settings. However, existing RAG systems still face several open challenges. Many approaches rely on static or heuristic retrieval strategies, limiting their ability to adapt retrieval strategies to different tasks with different reasoning requirements. Some methods build RAG systems based on independently optimized components, making joint optimization across all components infeasible. Moreover, empirical evidence shows that stronger retrievers do not necessarily lead to better end-to-end RAG performance, revealing a misalignment between retrieval quality and generation effectiveness and leaving considerable room for improvement in how retrieval is integrated into the RAG pipeline. Motivated by these challenges, we propose ARF-RAG, an Adaptive Retriever-Friendly Retrieval-Augmented Generation framework. ARF-RAG dynamically determines when to retrieve and further generates retriever-friendly queries that faithfully reflect the generator's current information needs while being optimized for the behavior of the given retriever. ARF-RAG adopts a role-unified mechanism, in which a single LLM simultaneously performs all retrieval-related and generation actions, including retrieval decision-making, query generation, and answer generation, enabling coherent optimization across all components. To effectively align the retriever and generator, we formulate RAG as a unified reinforcement learning (RL) problem. We design comprehensive reward functions that jointly account for answer correctness, retrieval effectiveness, and retrieval cost, enabling fine-grained policy optimization via proximal policy optimization (PPO). Extensive experiments. The codebase for reproducing the proposed framework is publicly available at: https://github.com/ii-research/ARF-RAG on three question-answering benchmarks demonstrate that ARF-RAG consistently outperforms strong non-RL and RL-based baselines on both simple and complex QA tasks. Further analyses show that ARF-RAG adaptively learns when to retrieve based on question complexity and generates more effective, retriever-aligned queries that resolve what to retrieve, highlighting the advantages of unified RL optimization for building robust and efficient RAG systems.