Toward System-Level Test and Evaluation of Agentic AI Systems: Assurance, Measurement, and Bounded Trust
Abstract
Agentic AI systems are moving from prototype demonstrations into operational workflows, yet test and evaluation (T&E) methods have not been defined to keep pace with this latest class of AI capabilities. Unlike stand-alone predictive or generative AI models, agentic systems plan multi-step work, execute actions, call tools, communicate with external services, maintain state, and hand tasks to other agents or humans. These capabilities make agentic systems useful in enterprise workflows, cyber operations, digital engineering, and decision-support settings, but they also make them harder to test and evaluate. This paper argues that T&E of agentic AI systems must move beyond model scoring to system-level assurance to establish operational confidence that a deployed agent will stay within its authority, recover from tool failures, preserve context correctly, reverse any harm it has caused, and leave enough evidence for a human to reconstruct what happened. We present a four-perspective framework for evaluating agentic AI systems: a systems perspective that defines the full system under test; a functional perspective that characterizes the degree of autonomy, authority, and workflow self-determination a system has within its operating environment; a risk perspective that addresses emergent multi-agent failure modes; and a measurement perspective that identifies technical, human-centered, temporal, and contextual metrics. We also outline a practical testing toolkit based on simulation, observational data, benchmarking, red teaming, and post-deployment monitoring.