Jul 2026· International Conference on Artificial Intelligence Testing· pp. 106-113· 0 citations· 42 references
Abstract
We propose a formal model for testing AI agents in stateful environments, using Finite State Machine (FSM) semantics to enable deterministic, trace-level evaluation via an explicit test oracle. Our approach introduces a classified action alphabet and six computable failure-mode detectors for automatic analysis over agent traces. Applying this methodology to 350 multi-turn network configuration runs, we find that aggregate scores often miss structural failures revealed by trajectory analysis, and that meltdown rates alone fail to distinguish distinct failure mechanisms. Agents satisfy locally verifiable intent properties more than twice as often as remotely verifiable ones, exposing a critical awareness–action gap (the difference in satisfaction rate between local and remote evidence). Trajectory-level analysis is therefore essential for robust evaluation of autonomous agents in complex systems.
LoopHarness is presented, which restores a persistent, non-decaying safety state at the loop level at the loop level, and gives a complete evaluation protocol on native Agent-SafetyBench tasks with paired clean and attacked episodes, an outer-state attack suite whose decisive evidence exists only across iterations, per...
Contextual security defenses prevent AI agents from taking rogue actions by synthesizing a task-specific policy and enforcing it on the agent's tool calls. In multi-step tasks, however, which actions are valid often depends on what the agent has already done and learned. We present Sapien, a policy engine for enforcing...
CO Tiffany, Wen Zhang, E. Bagdasarian et al.· 0 citations
This work presents ATLAS (Automata Learning for Agent Trajectory Analysis and Strategy Discovery), an approach for recovering interpretable behavioral models from agent trajectories that enable systematic understanding and analysis of otherwise opaque AI agents.
Ignacio D. Lopez-Miguel, A. Happe, J. Cito et al.· Proceedings of the ACM/IEEE...· 1 citation
It is argued harness engineering is a distinct, effective discipline for agent reliability, but its cost is model-dependent and must be measured, not assumed.
This work presents a post-hoc XAI framework that transforms a lengthy agent's execution trace into a structured report and a faithful natural-language explanation explicitly grounded in its observable behavior, outperforming naive LLM-generated explanations.
Vittoria Vineis, Fabiano Veglianti, Lorenzo Antonelli et al.· 0 citations
AgentRewind is presented, a runtime recovery framework that records aligned checkpoints of the agent context and controlled environment, allowing agents to return to an earlier state and resume execution with information from previous attempts, improving task success rate and average checklist progress over the compare...
Yu Zhuang, Kefei Chen, Yitong Duan et al.· 7 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
