Jun 2026· Proceedings of the ACM on Software Engineering· Vol 3, pp. 1888 - 1909· 1 citation· 50 references
TL;DR
FAMAS is the first spectrum-based failure attribution approach for MASs and proposes a novel suspiciousness formula tailored to MASs, which integrates two key factor groups, namely the agent behavior group and the action behavior group, to account for the agent activation patterns and action activation patterns within the MAS execution trajectories.
Abstract
Large Language Model Powered Multi-Agent Systems (MASs) are increasingly employed to automate complex real-world tasks, such as programming and scientific discovery. While promising, MASs are not immune to defects or failures. Failure attribution in MASs, i.e., to pinpoint the specific agent actions responsible for failures, is underexplored and labor-intensive, posing significant challenges for debugging and improving MASs. To bridge this gap, we propose FAMAS, the first spectrum-based failure attribution approach for MASs. The approach performs systematic trajectory replay and abstraction, followed by spectrum analysis. Its core idea is to estimate, from variations across repeated MAS executions, the likelihood that each agent action is responsible for the failure. In particular, we propose a novel suspiciousness formula tailored to MASs, which integrates two key factor groups, namely the agent behavior group and the action behavior group, to account for the agent activation patterns and action activation patterns within the MAS execution trajectories. Extensively evaluated against 12 baselines from the Who&When benchmark, FAMAS demonstrates superior performance, outperforming all compared methods.
AFANet is introduced, a lightweight graph-based framework that models interaction trajectories through step-level semantic signals and agent-level relationships and suggests that effective agent failure attribution does not require heavy LLM reasoning and a lightweight, structured approach can achieve strong performance.
Ting-Wei Li, Yuanchen Bei, Xiao Lin et al.· 0 citations
Adaptive Influence Graphs is introduced, a two-stage agentic framework that first transforms a failed trace into a structured graph and then navigates it to identify the critical error and establishes a new state of the art on Who&When, the standard benchmark for multi-agent failure attribution.
Yarden Bakish, Amir Dudai, Roy Ganz et al.· 0 citations
Experimental results show that AgentLocate consistently outperforms existing failure localization methods in identifying both responsible agents and failure steps, while remaining efficient in terms of token usage and running time.
Yu Xia, Anjun Gao, Yueyang Quan et al.· 0 citations
As large language model (LLM)-based multi-agent systems (MASs) are increasingly applied to long-horizon complex tasks, their reliability has emerged as the core bottleneck hindering their real-world deployment. Existing MAS debugging and repair methods typically rely on rerunning and resampling the entire execution trajectory. However, a fundamental question remains to be answered: do these methods causally repair MAS failures or merely stochastically repair by leveraging the randomness of LLM sampling? To evaluate the effectiveness of MAS repair methods, we introduce SymTrace, a controlled evaluation framework that records the MAS execution trajectory and establishes intervention anchors. During replay, it effectively reconstructs the execution before the anchor using recorded logs and only regenerates the downstream trajectory, thereby enabling the reliable reproduction of MAS failures. We further construct the dataset SymFail, comprising 536 human-annotated failure trajectories with graph-linked locations, categories, and trace evidence. Based on these foundations, we conduct a large-scale empirical study across three mainstream MAS frameworks. Our findings reveal that existing unguided rerun methods are highly unreliable, exhibiting low failure reproduction and repair rates (only 67.97% and 6.90%, respectively). Building upon these findings, we further explore the effectiveness of a symptom-driven intervention method, which successfully repairs 20.15% of the failed cases (a 191.89% improvement to state-of-the-art repair methods). This study aims to provide actionable insights for MAS debugging and repair research, paving the way for the robust deployment of multi-agent systems.
Zhong-Wen Luan, Xiaoyan Zhang, Ming Hu et al.· 2 citations
This work proposes Trajectory Graph Copilot, a novel framework that acts as a ``copilot'' for LLM agents by diagnosing potential action errors before they are executed, significantly enhancing the agent's ability to complete long-horizon tasks successfully.
Xu Zheng, Zhuomin Chen, Chaohao Lin et al.· 0 citations
Large Language Model (LLM)-based agents are increasingly used for complex tasks such as software testing and cybersecurity assessment. While these agents demonstrate impressive capabilities, their behavior is difficult to understand, explain, and analyze. Existing evaluations focus mainly on task success and execution traces, offering limited insight into the strategies employed by the agent. We present ATLAS (Automata Learning for Agent Trajectory Analysis and Strategy Discovery), an approach for recovering interpretable behavioral models from agent trajectories. ATLAS combines trace abstraction with automata learning to infer finite-state models that capture observed agent-environment interaction strategies. These models provide human-interpretable insights and support automated analyses of recurring behaviors, decision points, successful task-completion paths, and failure loops. As a proof of concept, we apply ATLAS to trajectories generated by an LLM-based penetration-testing agent. The resulting models expose high-level behavioral strategies for exploiting vulnerable machines that are difficult to identify from raw execution traces alone. We discuss how learned behavioral models can support explainability, model-guided exploration, auditing, and analysis of agentic systems. We further demonstrate symbolic model-based knowledge transfer from powerful frontier models to compact language models. In addition, we show how model transformations can derive concise explanations of agent behavior in a penetration-testing case study comprising 12 vulnerable machines. ATLAS highlights a new opportunity for model-driven engineering: transforming agent trajectories into explicit behavioral models that enable systematic understanding and analysis of otherwise opaque AI agents.
Ignacio D. Lopez-Miguel, A. Happe, Jürgen Cito et al.· 0 citations