Skip to content

Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery

Sep 2026 · 0 citations · 57 references
Computer Science

TL;DR

The account motivates complementary defenses: strengthen resistance and recovery alongside prevention of spontaneous deviations, and audit and restrict unintended communication paths that can turn local failures into collective loss of control.

Abstract

How does a multi-agent system evolve from a local deviation into collective loss of control? We propose an epidemic explanation organized around accidental mutation, contagion, and recovery. A spontaneous deviation creates a seed; communication enables other agents to adopt and retransmit its unsafe strategy; collective failure can emerge when propagation outpaces correction and containment. Thus, rare individual deviations can coexist with substantial collective risk. Motivated by reported OpenAI agent coordination incidents, we examine two ingredients of this mechanism. A deployment audit identifies implicit communication paths between nominally independent evaluation runs and verifies transport through a default Docker backend. RogueHandoff-20, a benchmark of 20 executable scenarios, tests recipient susceptibility by injecting unsafe trajectories generated by a modified Qwen-27B route. Across four native-pending routes, executed harm is 0-5% on normal tasks and 40-95% after injection, exceeding paired direct malicious requests by 5-45 percentage points. These results support low observed baseline harm alongside high conditional susceptibility; they do not establish natural rare-event rates or demonstrate an autonomous cascade. The account motivates complementary defenses: strengthen resistance and recovery alongside prevention of spontaneous deviations, and audit and restrict unintended communication paths that can turn local failures into collective loss of control.

View source

Similar papers

#machine learning Review Sep 2026

Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions

It is argued that the operational unit of defence should be a revisable coordination episode linking observed transfers, task authority, and response history, and it makes the recommendation to monitor across executions testable without claiming a new detector or a measured containment benefit.

Gregory N. Frank · 1 citation
Preprint Sep 2026

The Mechanics of a Swarm: A Reproducible External Reconstruction of an Unintended Agent-Coordination Episode on a Third-Party Wiki

Between 24 May and 2 July 2026, autonomous language-model agents running inside a timed research-question evaluation wrote to a third party's public, world-writable wiki. OpenAI acknowledged the incident; independent researchers reconstructed it and published the wiki's archived revision history. We analyse that histor...

Philipp Lütje · 0 citations
#artificial intelligence Preprint Sep 2026

Safety of Latent Communication in Multi-Agent Systems

Latent communication enables multi-agent systems to exchange information directly in internal representation space, reducing the token, computation, and latency overhead of text-based communication. To this end, lightweight trainable links are introduced to map the sender's representations into the receiver's input spa...

M. Huzaifa, Sina Mavali, Thorsten Eisenhofer · 0 citations
#artificial intelligence Preprint Aug 2026

COVER: Identifiable Evaluation of Coalition Routing

COLD is an auditable measurement methodology, an evaluation contract that fixes a public information boundary, downstream stack G, and finite legal team family before outcomes are generated, which exposes selection headroom without manufacturing a routing win.

R. Sugumar, Amrit Gopinath · 0 citations
#artificial intelligence Preprint Sep 2026

SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems

This work systematizes MAS security through an execution-centered analysis of 197 works, introducing an A-I-R framework that organizes attacks by adversary position, interaction interface, and resulting system-level risk, unifying otherwise fragmented attack mechanisms across MAS.

Rui Yang, Jun-Jie Xu, Zheng-Yu Liu et al. · 3 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.