It is shown that LLM-driven agents can violate this condition and introduced a canonical deployment wrapper that guarantees it for arbitrary base agents while preserving already-equivariant behaviour, and it is proved that computing canonical representations required by this construction is graph-isomorphism-hard.
Abstract
Agentic systems driven by large language models (LLMs) are increasingly deployed in real-world workflows where they act on persistent operational data. Before deployment, these systems need to be verified against business requirements that govern workflow execution and data evolution. However, existing approaches do not provide such system-level guarantees, as they mainly constrain or analyse behaviour at the agent's interface level. We study here the verification of agentic systems comprising a single LLM and a tool orchestration harness over relational operational data. We formalise them as Stateful Tool-Enabled Agentic Deployments (STEADs), give their semantics, define the problem of verifying them against First-Order Computation Tree Logic (FO-CTL) specifications, and show that it is undecidable. We identify sufficient conditions for exact preservation of FO-CTL specifications under a finite-domain restriction, over which verification is PSPACE-complete. The key requirement is that renaming opaque identifiers in the data must correspondingly rename the selected tool calls. We show that LLM-driven agents can violate this condition and introduce a canonical deployment wrapper that guarantees it for arbitrary base agents while preserving already-equivariant behaviour. We prove that computing canonical representations required by this construction is graph-isomorphism-hard. Finally, we illustrate our framework on an LLM agent orchestrating a case-management workflow.
Multi-agent systems built on large language models (LLMs) are increasingly deployed for complex tasks requiring autonomous planning, tool use, and inter-agent coordination. However, the non-deterministic nature of LLM outputs and the emergent behavior arising from agent interactions render traditional test oracles ineffective, creating a critical gap in quality assurance for agentic AI. This work introduces MORPHAGENT, a framework designed to address the oracle problem in multi-agent LLM systems through trace-based behavioral analysis. Our contributions are threefold: (1) goal-preservation relations that verify consistent goal achievement under input perturbations, (2) coordination-consistency relations that validate inter-agent delegation and communication patterns under agent substitution and reordering, and (3) tool-use integrity relations that ensure semantic equivalence of tool invocation sequences under prompt paraphrasing. MorphAgent instruments agent execution to capture structured traces comprising planning steps, tool calls, message exchanges, and final outputs, then systematically applies metamorphic transformations and checks behavioral invariants without requiring ground-truth oracles. We evaluate the framework on four multi-agent benchmarks spanning code generation, research synthesis, customer service, and data analysis tasks, encompassing 2,840 source-followup execution pairs across three LLM backends. Results show that MORPHAGENT detects 82.0% of seeded behavioral faults, including 90.3% of coordination failures and 81.7% of goal-deviation faults, while maintaining a false positive rate of 6.1%. The framework uncovers 14 previously unreported behavioral anomalies in established multi-agent frameworks, demonstrating its practical utility for assuring agentic AI reliability. These results suggest that trace-based metamorphic testing can serve as a practical foundation for reliable validation of emerging agentic AI systems.
Gopalakrishnan Marimuthu· International Conference on...· 0 citations
Deployed large language model (LLM) agents are now being used to interface with external tools, fetch information, run code, interact with user data and help with decision making at the workflow level. Therefore, their safety issues are not only related to the underlying model, but also to tool permissions, prompt design, retrieval sources, policy constraints, evidence validity, and varying operational contexts. While Goal Structuring Notation (GSN) has become a popular means of structuring safety arguments, many GSN-based safety cases are still reliant on manual inspection and can suffer from unsupported goals, incomplete evidence paths, undischarged assumptions, circular arguments, unaddressed defeaters, or insufficient coverage of hazards. This paper introduces MCSafe-GSN-HOL, a formal assurance framework for creating machine-checkable safety cases for deployed LLM agents that overcomes these disadvantages. The proposed framework translates the structured GSN arguments into Isabelle/HOL proof obligations, and formalizes the typed GSN elements, the hazard contracts of LLM-agents, the evidence-completeness conditions, the assumption-discharge rules, the defeater-resolution rules, and the incremental re-verification rules after deployment changes. The prototype was tested with 25 safety-case fragments from five representative scenarios of LLM-agents. Results demonstrate that MCSafe-GSN-HOL has 97.3% defect detection, 97.2% evidence coverage, 96.4% hazard-control coverage, and 62.7% reduction in review time compared to manual GSN review. The mean Isabelle/HOL checking time was 2.53 s/safety case, showing that theorem-prover supported assurance can still be practical for medium scale safety-case checking. The results overall support the conclusion that formally checked GSN safety cases can help with the traceability, auditability, consistency and maintainability of assurance processes for deployed LLM-agent systems.
Aakash Abhay Yadav, Shashank Shelat, B. Hinduja et al.· International Conference on...· 0 citations
: Integrating Large Language Models (LLMs) into production-grade business systems requires a clear boundary between deterministic application logic and non-deterministic AI behavior. This paper introduces a runtime architecture that extends the BAB (Business Application Builder) framework on Jakarta EE to support agent-augmented applications hosted alongside ontological-driven generated cores. The core of the solution, the AgentRuntimeManager , moves beyond simple API wrappers by providing a robust environment for agent lifecycles, asynchronous event dispatching, and error handling through configurable retry policies. To ensure system stability, we decouple LLM providers (such as OpenAI and Ollama) using an abstraction layer built on LangChain4j, allowing for runtime provider switching without redeploying the core logic. We demonstrate the practical utility of this architecture through a self-healing log-monitoring chain. In this setup, agents intercept high-severity log events and, through a structured analysis service, propose corrective actions. A specialized SqlErrorHealingAgent illustrates how LLM-proposed fixes can be filtered through layered safety constraints, such as the automatic blocking of destructive SQL operations like DROP or TRUNCATE, before any action is committed. The result is an architectural pattern that lets model-driven rigor and LLM agent flexibility coexist in enterprise environments without compromising either.
Borivoj Bogdanović, M. Segedinac, Z. Konjović et al.· SINTEZA· 0 citations
AgentFlow, a flow-centric policy language and runtime enforcement model for specifying where data may travel in agent systems, is presented and results are preliminary and scoped to the modeled policy-visible agent behaviors and evaluated benchmarks.
This work introduces the concept of an agentic transaction and proposes an ACID-compliant agent system framework that reinterprets the classical ACID properties for agent execution through four semantic guarantees: Semantic Atomicity, Semantic Consistency, Semantic Isolation, and Semantic Durability.
Agentao is presented, a governed local-first runtime for tool-using LLM agents that separates model-generated action proposals from host-authorized execution through a layered architecture consisting of host-facing surfaces, a host contract, a runtime core, a permission-mediated tool system, and supporting subsystems for memory, replay, plugins, skills, sub-agents, and protocol integration.
