Open Security Benchmark is presented, a framework that benchmarks agentic AI on security posture management work and surfaces a curated enterprise environment that evaluates posture investigation across two modalities: text-to-SQL over a relational snapshot and each vendor's native API over a served instance of the same environment.
Abstract
Enterprises are moving toward autonomous cyber defense: agentic AI that builds situational awareness of an organization's security state and reasons from it to assessments, decisions, and actions. This rests on a holistic view of the enterprise's security state, the continuous, cross-vendor picture of identities, cloud and infrastructure, data, applications, and their configurations that security posture management assembles. As agents take on this work, what matters is not whether an agent can produce an answer but whether it should be trusted to. The field cannot yet answer this question. Real enterprise environments are private, cross-vendor, and deeply correlated, and none is exposed publicly as a shared, queryable target for evaluating such agents end to end. We call this the environment data gap. We present Open Security Benchmark (OSB), a framework that benchmarks agentic AI on this work. OSB surfaces a curated enterprise environment - a frozen, holistic view of the security state - and evaluates posture investigation across two modalities: text-to-SQL over a relational snapshot and each vendor's native API over a served instance of the same environment. Freezing the environment pins the target state as an immutable snapshot and anchors answers to a closed-form ground truth. OSB is built from five components: a data layer, a task and evaluation-set layer, a multi-dimensional scoring layer, a minimal auditable harness, and a bring-your-own path that serves public comparison and private tenant evaluation from one substrate. We instantiate the framework with two identity-security packs and a family of synthetic-organization environment datasets spanning multiple scales, and chart its extension to further posture subdomains, investigation modalities, and defense stages from assessment toward remediation.
Cyber-capable AI agents combine language models with tools, memory, and execution environments to perform multi-step offensive-security tasks. Existing work separately measures cyber capability and catalogs attacks against agent components, but provides less guidance on containing a capable agent within the environments used to evaluate it. This review synthesizes five vulnerability classes at that boundary: multi-step offensive chains, objectives that conflict with sandbox boundaries, supply-chain and credential exposure, persistent command-and-control, and the speed of automated action. We use two separate preliminary incident records: the reported July 2026 Hugging Face/OpenAI evaluation breach and Anthropic's subsequent three-incident evaluation review. A comparative evidence protocol distinguishes record-specific factual claims from the shared systems lesson: the evaluation environment is itself part of the security boundary. Across the taxonomy and records, we examine controls for containment, privilege separation, provenance, and responder access, including the dual-use problem that defensive artifacts may also enable misuse. The review identifies practical priorities for evaluating cyber capability together with the security of the environment in which that capability is exercised.
Agent security is widely treated as a question about action content. Defenses ask whether an instruction looks malicious. Benchmarks ask whether an agent performs a harmful sounding action. \textbf{We argue that agent security is fundamentally a contextual problem, and that the current content based framing systematically misdefines it.} A command to ``delete user data''might be a routine administrative request or a prompt injection attacking production systems, and the content alone cannot distinguish the two. Authorization context can. Across every injection task in AgentDojo and WASP, the same action is one an authenticated user would plausibly request in a routine workflow, which makes the conflation a structural property of evaluating security through content. We operationalize contextual security through four properties that must hold jointly and be evaluated continuously across the agent's trajectory. Source Authorization asks who issued the command. Task Alignment specifies the agent's authorized objective. Action Alignment evaluates whether each action serves that objective. Data Isolation governs information flows across privilege boundaries. Under this reframing, indirect prompt injection becomes a Source Authorization violation. Snapshot benchmarks are structurally incapable of evaluating Data Isolation. Existing defenses are reorganized around the property they actually approximate. The contextual reframing changes which defenses are coherent, which evaluations measure something useful, and which attack patterns evaluation can see at all.
Vincent Siu, Jingxuan He, Kyle Montgomery et al.· 1 citation
Security in autonomous AI systems is not a property of individual components such as identity verification, static policy configurations, or model-level safeguards. Instead, it is a dynamic system property emerging from three interconnected dimensions: intent, time, and interaction. Intent captures the alignment between an operator’s goals and an agent’s behavior; time reflects the evolution of security states under adversarial pressure; and interaction describes how trust relationships between agents can propagate misalignment. This paper synthesizes the findings of the five-paper AI Agent Security Series into a unified, formal, and falsifiable theory of autonomous agent security. Current AI security approaches focus on component-level controls, including authentication, policy enforcement, and output filtering. However, results from Papers 1–5 show that these mechanisms are individually insufficient and collectively vulnerable to clean attacks semantically valid, policy-compliant adversarial inputs that redirect agent behavior while evading existing detection methods. We argue that such failures are structurally inevitable under component-centric security models. To address this gap, we introduce the Unified Security Function, Φ_security(φ, t, G), a scalar measure in the range [0,1] that combines IntentAlign, TemporalStability, and NetworkIntegrity into a single dynamic assessment of security health. Using this framework, we establish three meta-theorems: the Component Insufficiency Theorem, the Dynamic Necessity Theorem, and the Interaction Irreducibility Theorem. Evaluation on the complete AegisBench benchmark suite (1,560 sessions across Papers 1–5) shows that Φ_security predicts attack success with an AUC of 0.943, outperforming all individual metrics. By defining autonomous agent security as a distinct scientific object, this work provides a foundational framework for future research on secure and adaptive AI systems.
Harsh Verma· International Journal of Sci...· 0 citations
It is shown that the more consequential risks lie one layer down, in the protocol between agents and commerce services, and a platform-agnostic defense that drives the structural attack-success rate to zero for four of the five structural classes.
Cloud-native enterprise architectures have broken out the network perimeter, and now identity has become the front line of defense in the world of security. This review covers the current landscape of cloud-native enterprise identity and access management (IAM) solutions and how the industry has moved beyond a static approach to identity management and perimeter-based security, and is now shifting to dynamic, continually verified access control. A collection of essential research is synthesized covering access-control theory, federated authentication, security of containers and microservices, identity of the service, zero-trust concepts and machine-identity governance. On the basis of this synthesis, we argue that a four-plane theoretical model can be introduced to separate authentication, authorization reasoning, enforcement, and auditability into four closely coupled but independent evolving planes for the system. An illustrative evaluation, gleaned from published benchmarks, illustrates the expected behaviour of this model over 3 dimensions: authorization latency, enforcement overhead and scalability in the face of growth in identities. The results show that the model is viable for realistic enterprise workloads when optimizing policy evaluation and accounting for the overhead of policy enforcement, and provide limits for deployments using latency sensitive and throughput bounded workloads. Finally, the review highlights open challenges and directions for future research, providing a unified point of reference for both researchers and practitioners in the field of secure and scalable identity management in today's enterprise world.
Ravi Kumar Kotapati· International Research Journ...· 0 citations
AI agents are rapidly improving in cybersecurity capabilities when the source code is available for analysis, yet much of the software most consequential to cybersecurity, including malware, firmware, and proprietary applications, is available only as binaries. Analyzing such software requires reverse engineering(RE): recovering program semantics before the analysis can be meaningfully performed. However, evaluating agentic RE poses a fundamental challenge: benchmark instances must be unseen as source code in the LLMs'training data to prevent models from taking shortcuts by recognizing them rather than really analyzing them, while also matching the scale and anti-analysis protections of real software. Unfortunately, however, existing benchmarks do not jointly satisfy these requirements. To this end, we introduce SRE-Bench, the first realistic, contamination-free RE benchmark. Built entirely from scratch by RE experts with over 5,000 hours, SRE-Bench comprises 19 private, real-world-scale programs averaging 16.9K lines of code. We further developed 44 in-house anti-analysis primitives, yielding 262 binary instances and 1572 deterministically graded tasks. Our evaluation across five frontier LLMs (GPT-5.6-sol,Claude-Opus-5,GPT-5.5,Grok-4.5, and GLM-5.2) shows that RE remains largely unsolved: the strongest model, GPT-5.6-sol, scores 61.4% per instance, and fully solves only 31.5% of the instances. Our analysis further reveals that agents behave differently from human engineers, where agents are relatively insensitive to compiler optimization and static linking. Controlled ablations also confirm that both contamination control and realistic scale are essential. These results indicate that strong source-code security capabilities do not yet transfer to binary analysis, highlighting RE as an important frontier for agentic cybersecurity and SRE-Bench as a rigorous testbed to measure progress.
J. Spence, Nicholas Assaderaghi, Jinhao Zhu et al.· 0 citations