DFRL-Mol: a dual-stage framework of reinforcement learning for multi-scenario molecule optimization

Abstract

Abstract As a critical step in drug discovery, molecule optimization aims to improve specific properties of lead compounds. Inspired by isosteres (i.e. molecular substructures with similar reactive electron shells), existing AI-based methods have been proposed for molecule optimization. However, they often rely on available matched molecular pairs (MMPs), thus facing two essential challenges. First, the limited diversity of available MMPs leads to biased training and a lack of novelty in the optimized molecules, in both single-objective (SO) and multi-objective (MO) optimization. Second, the low number of available MMPs leads to limited generalization in MO optimization. To overcome the limitations of MMPs, we propose DFRL-Mol, a novel dual-stage reinforcement learning framework. It consists of three main modules: the Structure–Property Analyst (SPA), the Decorator (DCR), and Curriculum-guided Dual-Stage Policy Optimization (CDSPO). SPA and DCR are two LLM-based models, which are responsible for locating the substructures to be optimized and inferring their appropriate isosteric replacement, respectively. CDSPO leverages curriculum-guided reinforcement learning to facilitate collaboration between the two models. DFRL-Mol ensures the generation of diverse molecular representations while effectively overcoming the issues of bias and data sparsity associated with MMPs. The comparisons with state-of-the-art methods demonstrate the enhanced performance of DFRL-Mol in multiple scenarios of molecule optimization, including SO optimization, substructure-constrained (SC) SO optimization, MO optimization, and MO molecule generation. More importantly, detailed analysis of experimental results demonstrates that the models’ behavior is consistent with chemical intuition.