Survival-aware self-play reinforcement learning for partially observable beyond visual range air combat

Abstract

We present a recurrent self-play reinforcement learning framework for autonomous beyond-visual-range (BVR) air combat under partial observability. The environment is formulated as a decision-level BVR missile-duel simulator that combines 3-DoF aircraft kinematics, field-of-view radar observation, track-quality decay, flight-envelope action masking, and an energy-aware missile model with boost, coast, drag, and proportional-navigation guidance. To handle intermittent radar contact, the policy uses an LSTM backbone. To stabilize recurrent training, we introduce trajectory-aligned hidden-state management and mask recurrent sequences at episode boundaries. The training follows an asymmetric self-play scheme in which the protagonist is updated by a recurrent MAPPO-style objective with centralized value input, while opponents are sampled from frozen historical policy snapshots. A survival-aware reward function balances offensive alignment, radar tracking, beam-like defensive maneuvering, missile-threat avoidance, and time cost. Experiments and ablations use episodic reward as a process-level training indicator and win rate as the final outcome-level confrontation metric. The results show that recurrent memory, opponent-pool self-play, and survival-aware reward shaping improve robustness in partially observable BVR engagements. The current study focuses on controlled 1-vs-1 BVR missile duels and provides a basis for future N-vs-N cooperative extensions.