Charting these challenges provides a roadmap toward trustworthy autonomous agent deployment: security must become a verifiable property of the architectures, protocols, and runtimes that govern agent behavior, rather than an optional layer of guidance.
Abstract
Autonomous agents are increasingly used to execute consequential tasks in environments governed by operational constraints, organizational policies, regulatory requirements, and technical standards. Their safety is therefore determined not by the correctness of individual actions, but by whether their overall behavior remains consistent with the rules and invariants of the systems in which they operate. As large language model (LLM)-based agents become more autonomous and increasingly delegate tasks across organizational boundaries, securing them evolves from a single challenge into a broad and interconnected landscape spanning the entire agentic stack. At the single-agent level, untrusted inputs through prompts, memory, retrieved knowledge, and tool interfaces create attack surfaces. In multi-agent settings, delegation and communication introduce challenges related to identity, trust, capability control, and decision transparency, while the underlying model routing and execution control plane remains vulnerable to manipulation and to unverified model provenance. Perhaps the most fundamental challenge is behavioral containment: sequences of individually permissible actions may collectively violate system-level constraints and safety invariants. At the broader level, supply-chain integrity, provenance, accountability, and end-to-end observability remain largely open problems. A common principle unifies these directions: security must become a verifiable property of the architectures, protocols, and runtimes that govern agent behavior, rather than an optional layer of guidance. Charting these challenges provides a roadmap toward trustworthy autonomous agent deployment.
EvidenceNet is presented, a runtime assurance layer for deciding whether coordinated agent operations have achieved an operator's network intent, and experiments on live routing networks show that post-change state checks recognize successful outcomes that configuration-action records alone cannot establish.
This is the first paper from POLIS, an ongoing research programme studying algorithmic institutions for multi-agent systems, and asks which parts of an AI institution produce safety and how they do it.
OpenAgentFlow is presented, a control-plane/action-plane architecture that establishes the action-commit boundary as a shared enforcement interface and shows that a shared action-commit boundary provides a practical basis for system-wide governance across heterogeneous agent execution paths.
Dong-Sheng Chen, Xiang-Yu Zhao, Xin Yao et al.· 0 citations
This paper proposes two complementary artifacts: an Agency Justification Record (AJR) helps teams decide when an agent is warranted over simpler alternatives and an Agentic Delegation Policy (ADP) captures what must be specified for safe and effective development.
Chetan Arora, Andreas Vogelsang, Abbishek Sharma· arXiv.org· 0 citations
Agentic networks transform accepted intents into operational services through autonomous reasoning, adaptive planning, tool use, and cross-domain coordination, but these capabilities introduce failure modes that conventional reliability measures do not fully capture. An accepted intent may still be carried out incorrec...
Bilgehan Erman, Andrea Francini, Nikos Papadis· arXiv.org· 0 citations
Synthesizing across four constraint-bound engineering domains, agentic AI trustworthiness is shown to be a single problem, with a path outlined toward a reusable, cross-domain assurance framework analogous to the graded certification regimes used by mature safety-critical engineering fields.
Omar Al-Refai, Ibrahim Shahbaz, A. Husseinat et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.