Reconceptualizing Observability for Agentic AI Systems: A Trace-Centric Architecture for Interpreting Non-Deterministic Workflow Behavior

Metamorphic Testing of Multi-Agent LLM Systems: A Trace-Based Behavioral Oracle Framework

Multi-agent systems built on large language models (LLMs) are increasingly deployed for complex tasks requiring autonomous planning, tool use, and inter-agent coordination. However, the non-deterministic nature of LLM outputs and the emergent behavior arising from agent interactions render traditional test oracles ineffective, creating a critical gap in quality assurance for agentic AI. This work introduces MORPHAGENT, a framework designed to address the oracle problem in multi-agent LLM systems through trace-based behavioral analysis. Our contributions are threefold: (1) goal-preservation relations that verify consistent goal achievement under input perturbations, (2) coordination-consistency relations that validate inter-agent delegation and communication patterns under agent substitution and reordering, and (3) tool-use integrity relations that ensure semantic equivalence of tool invocation sequences under prompt paraphrasing. MorphAgent instruments agent execution to capture structured traces comprising planning steps, tool calls, message exchanges, and final outputs, then systematically applies metamorphic transformations and checks behavioral invariants without requiring ground-truth oracles. We evaluate the framework on four multi-agent benchmarks spanning code generation, research synthesis, customer service, and data analysis tasks, encompassing 2,840 source-followup execution pairs across three LLM backends. Results show that MORPHAGENT detects 82.0% of seeded behavioral faults, including 90.3% of coordination failures and 81.7% of goal-deviation faults, while maintaining a false positive rate of 6.1%. The framework uncovers 14 previously unreported behavioral anomalies in established multi-agent frameworks, demonstrating its practical utility for assuring agentic AI reliability. These results suggest that trace-based metamorphic testing can serve as a practical foundation for reliable validation of emerging agentic AI systems.

Chasing Abpms Vision: The Bab Framework Approach for LLM Agents Over Model-driven Enterprise Systems

: Integrating Large Language Models (LLMs) into production-grade business systems requires a clear boundary between deterministic application logic and non-deterministic AI behavior. This paper introduces a runtime architecture that extends the BAB (Business Application Builder) framework on Jakarta EE to support agent-augmented applications hosted alongside ontological-driven generated cores. The core of the solution, the AgentRuntimeManager , moves beyond simple API wrappers by providing a robust environment for agent lifecycles, asynchronous event dispatching, and error handling through configurable retry policies. To ensure system stability, we decouple LLM providers (such as OpenAI and Ollama) using an abstraction layer built on LangChain4j, allowing for runtime provider switching without redeploying the core logic. We demonstrate the practical utility of this architecture through a self-healing log-monitoring chain. In this setup, agents intercept high-severity log events and, through a structured analysis service, propose corrective actions. A specialized SqlErrorHealingAgent illustrates how LLM-proposed fixes can be filtered through layered safety constraints, such as the automatic blocking of destructive SQL operations like DROP or TRUNCATE, before any action is committed. The result is an architectural pattern that lets model-driven rigor and LLM agent flexibility coexist in enterprise environments without compromising either.

Supporting Autonomous Process Execution within a Multi-Perspective Constraint Frame via Numeric Planning

AI-Augmented Business Process Management Systems (ABPMS) enhance traditional BPMS by leveraging advanced AI techniques to define, execute, and monitor complex process structures. Within this landscape, Framed Autonomy denotes the capability of a system to autonomously advance the execution of a Business Process (BP) instance while strictly adhering to a predefined frame, i.e., a set of constraints that may span multiple perspectives. Existing research on framed autonomy has predominantly focused on control-flow constraints, either declarative or procedural, and typically relies on their transformation into automata-based representations. In this study, we extend this line of work by introducing a novel tool for what-if analysis that augments the process frame with multi-perspective constraints, including data-aware and temporal conditions. Given a partial process execution, the proposed approach exploits this enriched frame to recommend optimal continuations in compliance with the underlying process specifications. We additionally report an empirical evaluation demonstrating the scalability and effectiveness of the technique, thereby highlighting its potential for supporting autonomous and constraint-aware decision making in ABPMS.

Agent-in-the-Loop: Using AI Agents to Perform Control-Oriented Simulation Tasks

The Evaluation Context Protocol (ECP): A Portable Contract for AI Agent Evaluation

This paper proposes the Evaluation Context Protocol (ECP), an early-stage, vendor-neutral framework intended to act as a portable evaluation contract layer for agentic systems and describes an open-source reference implementation that includes adapters for LangChain, LlamaIndex, CrewAI, and PydanticAI.

A Systematic Survey of LLM-Based Agentic AI Frameworks for Multi-Agent Coordination and Interoperability

Large Language Models (LLMs) have spurred the development of agentic artificial intelligence systems that can reason autonomously, plan, use tools, integrate memory, and carry out multi-step tasks. Unlike conventional prompt-response systems, LLM-based agents extend generative models to goal-oriented architectures that can decompose complex objectives, interact with external environments, and coordinate actions in iterative workflows. The survey systematically reviews LLM-based agent frameworks, multi-agent collaboration architectures, internal communication mechanisms, emerging cross-system interoperability protocols, and open research challenges. The review uses a PRISMA-based methodology, including literature from 2020 to March 2026, with particular focus on prominent frameworks such as LangChain, AutoGPT, AutoGen, MetaGPT, CAMEL, ChatDev, and CrewAI. The survey discusses the transition from modular chain-oriented to stateful graph-oriented and autonomous execution models and presents a systematic taxonomy of multi-agent collaboration architectures, including hierarchical, peer-to-peer, and role-based models. It also reviews the main internal communication mechanisms used to facilitate coordination and information sharing between LLM-based agents and distinguishes them from emerging interoperability protocols designed to support interaction across heterogeneous agents, tools, services, and frameworks. The analysis indicates that the promise of LLM-based agents for scalable automation, collaborative reasoning, and complex workflow execution comes with significant challenges in long-horizon reliability, evaluation standardization, communication security, cost-efficient orchestration, governance, and the interpretability of emergent multi-agent behavior. The survey covers architectural evolution, collaboration patterns, communication mechanisms, framework-level characteristics, and open research challenges. This provides a structured foundation for future research on reliable and trustworthy agentic artificial intelligence systems.