Balancing adversariality and naturalness in autonomous vehicle testing scenario generation via multi-objective reinforcement learning

Abstract

Reinforcement learning (RL)-based adversarial scenario generation through dynamic agent–environment interactions has emerged as a promising approach for accelerating autonomous vehicle (AV) testing. However, existing RL approaches struggle to balance adversariality and naturalness because naturalistic driving behaviors avoid risk whereas adversarial behaviors intentionally induce it. The reward-driven nature of RL can further lead to objective polarization under linearly weighted rewards, resulting in scenarios biased toward a limited range of adversariality–naturalness trade-offs. This study proposes a preference-conditioned safety–critical scenario generation framework based on multi-policy multi-objective reinforcement learning (MORL), which models adversariality and naturalness as separate components of a vectorized reward and jointly optimizes them across the preference space. The adversariality reward incorporates risk–liability screening to filter two predefined invalid-risk patterns, while the naturalness penalty leverages a computationally efficient shadow-mode evaluation to provide real-time behavioral-deviation feedback relative to a reference driving policy. Envelope Q-learning is adopted to learn multiple preference-conditioned policies and is enhanced by a Kullback–Leibler divergence–based policy differentiation mechanism that mitigates policy mode collapse and encourages diverse behavioral evolution. Experiments in highway car-following and lane-changing environments show improvements in the evaluated scenario-quality and diversity metrics over the RL baselines. Compared with multi-round linearly weighted RL, the proposed method achieves 2.70 times the Pareto-front hypervolume, increases the valid safety–critical scenario success rate from 4.4% to 14.6%, and achieves 2.83 times the feature-space coverage volume. The results support the framework’s ability to generate diverse safety–critical scenarios across testing preferences within the studied highway operational design domain.