Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation
TL;DR
This work introduces Evolutionary Safety as a perspective for studying safety under persistent and recursive self-improvement, which concerns not only whether an AI system is safe at a particular moment, but how safety properties change, persist, accumulate, and propagate throughout evolution.
Abstract
Rapid advances in AI are bringing recursive self-improvement (RSI) closer to practice, with growing progress in model self-training, self-evolving agents, and automated AI development. Yet the very mechanisms that enable continuous improvement also introduce new challenges for AI safety. Unlike isolated failures, safety risks can persist across updates, accumulate through repeated adaptation, and propagate to successor systems. More importantly, the processes that generate and select future improvements may themselves evolve, making safety increasingly difficult to assess and maintain. This raises a fundamental question, i.e., how can safety be maintained when both AI systems and the processes that improve them continue to change? Therefore, we introduce Evolutionary Safety to study how safety properties are preserved, weakened, or restored throughout persistent and recursive self-improvement. We characterize six recurring manifestations of evolutionary risk and develop a five-domain taxonomy that captures how safety-relevant changes arise, persist, and propagate across evolving systems. Building on this foundation, we establish a framework for risk discovery and evaluation across states, updates, trajectories, and lineages, and examine governance principles for controlling modifications, verifying and authorizing updates, and ensuring traceability and recovery. We further identify open problems in maintaining safety as both AI systems and their improvement mechanisms continue to evolve.
