RACE-SOC: Risk-Constrained Auditable Coordination Evaluation for LLM-Based Multi-Agent Security Operations

Abstract

: Large language model (LLM) agents are now being used in security operations center (SOC) workflows to summarise security alerts, support triage, and suggest escalation or containment measures. Existing evaluations tend to focus on answer quality or benchmark accuracy, while the operational risks of multi-agent coordination are not clearly exposed. This gap is problematic in SOC triage because extra agents may produce unsupported claims, disagreement, false consensus, or unsafe operations even when the final answer seems reasonable. This paper proposes RACE-SOC, a Risk-constrained Auditable Coordination Evaluation protocol for LLM-based multi-agent SOC triage. RACE-SOC evaluates bundled role-conditioned configurations over fixed source groups by scoring the trade-off among triage utility, coordination risk, and token cost. The corrected deterministic scorer combines utility terms with a K-invariant primary risk score based on bounded unsafe-output incidence and unsupported-claim rate; coordination-only disagreement and false-consensus terms are reported separately. Across 45 model–dataset–seed cells, the corrected operational selections are K = 1/5/10 in 13/14/18 cells, including seven K = 5/K = 10 co-best cells resolved by a preregistered cost-first tie-break. Each sampling seed realizes n = 28 source groups (6/6/4/6/6 across the five datasets), giving 84 seed-indexed observations but only 54 distinct alert-set signatures with overlap; these are not statistically independent population samples. We instantiate the protocol on five public cybersecurity datasets and compare Qwen3.5-9B local inference, GLM-5.2, and DeepSeek-v4-pro. We also add a matched same-temperature repeated-inference control at K = 5 and K = 10 to separate repeated sampling from the role-conditioned pipeline. The control obtains a higher mean net score in five of six model–K strata, while the role-conditioned pipeline is higher for GLM-5.2 at K = 10. The direction of all six comparisons is retained after unique-signature weighting and after excluding plurality ties. These mixed results do not support a general benefit from role specialization under the tested design. Instead, they show why agent count, prompt view, aggregation, and risk must be audited together. The contribution of RACE-SOC is a reproducible evaluation instrument for exposing these configuration boundaries in SOC triage rather than assuming that more roles are inherently better.