On AI operational alignment and misalignment for high-risk AI systems: case study

Abstract

AI alignment is generally associated with ethics and social aspects. For high-risk AI systems, principles such as safety, stability and performance are crucial for their adoption in real-world application, to build trust and increase efficiency of a process. By safety it is meant both safety of the system itself but also safety of the process and environment. Any decision taken by a high-risk AI system should preserve safety, ensure continuous operation, real-time functioning, smooth control, and improve process efficiency. Human oversight needs to be continuously ensured, both to preserve safety of the process in case of transition from autonomous to manual mode in case of failures, but also to allow for contextual knowledge of the decisions. All these objectives are included in the AI operational alignment. High-risk AI systems designed to automatize complex processes in critical environments are continuously adapting to dynamic contexts, thus a static alignment might not be sufficient to properly assess their behaviour. The paper describes an approach to automated dynamic operational alignment in a complex high-risk process, exemplified by a case study on autonomous drilling. Possible sources of AI misalignments in this case are discussed and their potential implications.