The paper characterizes when agentic mechanisms change system behavior, whether these changes improve task completion, and how the observed failure modes motivate an optimized agent design under the same evaluation harness.
Abstract
Large language model (LLM) agents are increasingly used for complex information-extraction tasks, yet it remains unclear whether agentic components such as reflection and memory lead to observable and controllable improvements over fixed LLM workflows. We study this question through conference-paper dataset extraction, where a system must identify datasets mentioned in scholarly PDFs and produce structured records. We compare a fixed workflow baseline with reflective agent variants and specify an optimized agent condition (S2) that extends the same task with richer PDF tools and dynamic tool selection. Our evaluation emphasizes process-level behavior--including tool execution, retries, reflection, memory use, runtime, and failure recovery--while treating extraction coverage and field completeness as secondary outcome measures. The paper characterizes when agentic mechanisms change system behavior, whether these changes improve task completion, and how the observed failure modes motivate an optimized agent design under the same evaluation harness.
EvoSOP is introduced, a framework that empowers agents to extract SOPs from execution trajectories and iteratively optimize the toolset through a systematic lifecycle of construction, merging, evaluation, and pruning, providing a scalable pathway for the development of self-evolving agents.
Haipeng Ding, Yuexiang Xie, Zhewei Wei et al.· 2 citations
Large language model (LLM) agents are increasingly used for multi-step, stateful tool-use tasks, yet production reliability remains limited. Unlike static software repair, agent repair must recover dynamic trajectories whose early decisions can propagate into later errors and external state changes. Existing automatic remedies address only part of this problem: blind retry adds no diagnosis, outcome feedback says whether a run failed but not where or why, and self-reflection often lacks grounded evidence to prevent the same failure from recurring. We present AgentTether, a run-time repair framework that automates post-run diagnosis and guided recovery without modifying the underlying agent or environment. AgentTether abstracts each run into Transition Units, links them through a dependency-aware Critical Transition Graph, and localizes failure-critical subtrajectories by combining an offline normal-behavior model with a run-local graph detector. It then converts the localized cause into behavior-scoped guidance backed by cross-iteration Repair Memory, and can optionally apply guarded run-time intervention to keep the correction active during re-execution. The same design can be deployed as an offline diagnostic-and-guidance tool or as an online repair layer. We evaluate AgentTether on 261 tau-bench tasks across three domains with Qwen3.7-max, and test cross-model transfer on Banking with GPT-5.4. On the hardest Banking domain, AgentTether repairs 59.04% (49/83) of initially failed Qwen3.7-max tasks and 65.12% (56/86) of initially failed GPT-5.4 tasks. Overall, AgentTether improves repair effectiveness while reducing agent turns and end-to-end approach tokens, suggesting a practical reliability layer that can wrap existing agent deployments, reduce wasted re-execution, and improve recovery without retraining the agent.
Chenyu Zhao, Shenglin Zhang, Wenwei Gu et al.· 2 citations
Multi-agent systems built on large language models (LLMs) are increasingly deployed for complex tasks requiring autonomous planning, tool use, and inter-agent coordination. However, the non-deterministic nature of LLM outputs and the emergent behavior arising from agent interactions render traditional test oracles ineffective, creating a critical gap in quality assurance for agentic AI. This work introduces MORPHAGENT, a framework designed to address the oracle problem in multi-agent LLM systems through trace-based behavioral analysis. Our contributions are threefold: (1) goal-preservation relations that verify consistent goal achievement under input perturbations, (2) coordination-consistency relations that validate inter-agent delegation and communication patterns under agent substitution and reordering, and (3) tool-use integrity relations that ensure semantic equivalence of tool invocation sequences under prompt paraphrasing. MorphAgent instruments agent execution to capture structured traces comprising planning steps, tool calls, message exchanges, and final outputs, then systematically applies metamorphic transformations and checks behavioral invariants without requiring ground-truth oracles. We evaluate the framework on four multi-agent benchmarks spanning code generation, research synthesis, customer service, and data analysis tasks, encompassing 2,840 source-followup execution pairs across three LLM backends. Results show that MORPHAGENT detects 82.0% of seeded behavioral faults, including 90.3% of coordination failures and 81.7% of goal-deviation faults, while maintaining a false positive rate of 6.1%. The framework uncovers 14 previously unreported behavioral anomalies in established multi-agent frameworks, demonstrating its practical utility for assuring agentic AI reliability. These results suggest that trace-based metamorphic testing can serve as a practical foundation for reliable validation of emerging agentic AI systems.
Gopalakrishnan Marimuthu· International Conference on...· 0 citations
SyntheticAgentTraceQA is proposed, an execution- first framework for generating scalable supervision data for tool- augmented agents and shows that execution-grounded supervision improves tool execution behavior, reference-trace agreement, and answer-generation performance on the evaluated tasks.
Hafsa Ouajdi, Francesco Giannuzzo, Alaa Boukhary et al.· 1 citation· ⚡1
Recent advances in large language models have enabled a new class of agentic data science systems that allow users to complete complex data science workflows through natural language. Although these systems can significantly reduce manual effort, it remains difficult to diagnose their behavior and steer the reasoning process when failures or unexpected outputs occur. We present MUSE, an interactive meta-agent that enhances user understanding and control of agentic data science systems by (1) dynamically restructuring low-level execution traces into multiple semantic levels that support navigation from high-level overviews to low-level implementation details; (2) enabling users to reference specific workflow steps in context to ask grounded questions, provide feedback, and revise problematic steps without manually locating relevant execution history; and (3) supporting mixed-initiative steering by surfacing suspicious steps for inspection, scaffolding the repair process, and translating user repair intent into contextualized instructions for the underlying agent. In a between-subjects study (n = 15), MUSE improved task efficiency and increased users'confidence in understanding and steering agentic data science workflows.
Wei-Hao Chen, Weixi Tong, Yuan Tian et al.· 0 citations
The more typical feature of agentic AI systems is dynamic, multistep workflows where autonomous components plan, reason, and communicate with external tools and data sources in a series of iterations. Such flexibility increases capability but also brings nondeterminism which is inherent and where the same inputs can result in different execution paths and outputs. The variability creates a major challenge to the traditional observability approaches that are mostly created to support deterministic and service-oriented architectures. This paper redefines observability as an architectural element and introduces a trace-oriented architecture to suit agentic processes. The suggested solution presents semantically rich trace units capturing reasoning transitions, the intent to select a tool, the evolution of memory, and policy interactions, to gain a better insight into the execution behaviour. One of the major innovations is the combination of branch-aware trace modelling with an adaptive fidelity mechanism to dynamically change monitoring granularity based on uncertainty and anomaly indicators. Experimental analysis of various agent workflows shows significant increases in completeness of traces, accuracy in detection of anomalies and localization of root-cause, and a significant decrease in diagnosis time. The findings suggest that the suggested architecture does not only increase interpretability but also helps to ensure the reliable and efficient functioning of non-deterministic AI systems.
Ankur Gupta, Karan Gupta, Divyakumar Deepak Savla et al.· International Conference on...· 0 citations