Connecting Uncertainty Quantification to Safety Engineering: Uncertainty Gate Framework for Safe ML Decision Making

Abstract

Deploying Machine Learning (ML) models in safety-critical domains requires connecting their behavior to safety engineering standards such as IEC 61508. These standards focus on mitigating systematic failures in software, yet ML models also exhibit failures that are better treated as random and thus fall outside this scope. Uncertainty quantification can characterize such failures through a per-prediction uncertainty score, but a score alone does not determine whether an output is safe; this requires an additional step, referred to as uncertainty handling.This work introduces Uncertainty Gate, an uncertainty-handling framework that is agnostic to both the underlying ML model and the source of uncertainty. The framework treats the uncertainty score as an online diagnostic that accepts or rejects each prediction against a threshold. It then connects the resulting risk to IEC 61508 through a finite-sample bound, which accommodates dependent (non-i.i.d.) test data via a proper cover number. This allows practitioners to determine whether a given dataset and model meet a target Safety Integrity Level (SIL), and how much test data such a guarantee requires. We demonstrate the framework on a worked example targeting SIL 1 and discuss its use cases, extensions, and limitations.