Skip to content

MAADBench: The Refreshable Paradigm for Anomaly Detection in Multi-Agent Systems

Sep 2026 · 0 citations · 84 references
Computer Science

TL;DR

MAADBench is presented, the first refreshable MAS AD benchmark designed for diverse, evolving LLM backbones underlying the agents, and automated provision of cost-free, deterministic step-level labels for fine-grained AD evaluation.

Abstract

Recent studies report that LLM-based multi-agent systems (MAS) fail at rates of 41%-87%, yet to our knowledge, no benchmark to date supports systematic anomaly detection (AD) for them. Building MAS AD benchmarks is hard because they must remain fresh as LLM systems evolve: tasks may leak into training data and thus be memorized by LLMs, traces and anomaly patterns expire as backbones evolve, and labels must be provided reliably for each refresh. To address these challenges, we present MAADBench (MA: multi-agent; AD: anomaly detection), the first refreshable MAS AD benchmark designed for diverse, evolving LLM backbones underlying the agents. MAADBench combines (1) sampled-and-coupled generative tasks over an approximately 10^37-task space to mitigate task leakage, (2) refreshable trace generation under configurable LLM backbones, and (3) automated provision of cost-free, deterministic step-level labels for fine-grained AD evaluation. Beyond offering the paradigm itself, we run MAADBench with five state-of-the-art LLM backbones and release the MAADBench-Full dataset with 5,200 step-labeled traces. Benchmarking 25 AD methods on the MAADBench dataset reveals substantial limitations in current approaches: they rely heavily on supervision, struggle with subtle MAS-specific anomalies, and lack robustness across LLM backbones. These gaps point to a rich research agenda for MAS-specific anomaly detection, with MAADBench providing a systematic and refreshable testbed for method development and evaluation. We open-source MAADBench-Full at https://huggingface.co/datasets/hww123/MAADBench-full.

View source

Similar papers

Book Open access Oct 2026

AgentSysFilter: A Multi-agent Analysis Approach for Scenario-Aware Syscall Limitation

Modern OS kernels expose hundreds of syscalls to applications, yet individual programs typically require only a small subset. This over-privileging violates the principle of least privilege and expands the kernel attack surface. Existing methods for limiting system calls face several challenges: dynamic tracing suffers...

D. Zhan, Xing-Chen Yan, Zhao Yu et al. · 0 citations
Preprint Aug 2026

Automata from Agent Traces: Failure and Next-Step Prediction

Behavioral topology is shaped more by the deployment harness than by the LLM, providing a model-agnostic structural primitive for safety auditing and runtime monitoring, and addresses both prediction goals.

Seonglae Cho, F. Fernandez, Umar Mohammed et al. · 4 citations · ⚡1
Preprint Aug 2026

From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

This work presents AgentSysBench, a benchmark suite and measurement toolkit with ten representative agentic applications and unified systems-level instrumentation, and identifies six properties that distinguish agentic workloads from conventional LLM serving.

Chaokun Chang, Yu-Kun Zhou, Kai-Hua Fu et al. · 11 citations
Preprint Aug 2026

Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems

This study introduces SymTrace, a controlled evaluation framework that records the MAS execution trajectory and establishes intervention anchors and explores the effectiveness of MAS repair methods, revealing that existing unguided rerun methods are highly unreliable.

Zhong-Wen Luan, Xiaoyan Zhang, Ming Hu et al. · 2 citations

Smarter by the Moment: Environment-Driven Dynamic Policies for Continual LLM Improvement

Dynamic Retrieval-based Policy Generation (DRPG) is proposed, a framework that integrates memory-based retrieval with a dynamic policy generator, leveraging historical data and environment feedback to produce task-specific policies for continual LLM improvement.

Ting-Wei Chang, Po-Chun Chen, Hen-Hsen Huang et al. · 0 citations
Sep 2026

Scalable and Adaptive Log-based Anomaly Detection: A Synergistic Approach

SeaLog is presented, an accurate, lightweight, and adaptive log-based anomaly detection framework that synergistically integrates the efficiency of traditional data-driven methods with the zero/few-shot learning ability of large language models (LLMs).

Jin-Yang Liu, Jun-Jie Huang, Zhihan Jiang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.