POLO: Preference-Guided Multi-Turn Reinforcement Learning for Sample-Efficient Lead Optimization

Abstract

Lead optimization in drug discovery requires iteratively refining molecular candidates while preserving structural similarity to the original compound. Since each evaluation is costly, sample efficiency, the ability to achieve strong performance with limited oracle calls, becomes critical. Existing methods, from genetic algorithms to LLM-based approaches, rarely leverage intermediate outcomes and oracle feedback within an optimization trajectory, which reduces sample efficiency and forces more oracle calls to reach high success rates. To address this, we present POLO (Preference-guided Optimization for Lead Optimization), a multi-turn Reinforcement Learning framework for sample-efficient lead optimization. At its core, POLO introduces Preference-Guided Policy Optimization (PGPO), which extracts learning signals from each trajectory at two complementary levels: trajectory-level policy optimization reinforces successful multi-turn strategies, while turn-level preference learning ranks intermediate molecules by oracle-derived rewards and constructs within-trajectory pairwise comparisons to provide denser learning signals for effective modifications. Crucially, these preference signals require no additional oracle calls, as they are derived from evaluations already collected during rollouts. Experiments demonstrate that POLO achieves 85% success rate on single-property tasks (40% improvement over best baselines) and 50% on multi-property tasks using only 500 oracle calls. Our code is available at https://github.com/REAL-Lab-NU/POLO.